{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rent Listing Inqueries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rental Listing Inquiries数据集是Kaggle平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征x共有14维，响应值y为用户对该公寓的感兴趣程度。评价标准为logloss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.参数调优（细调）：n_estimators  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"RentListingInquries_FE_train.csv\")\n",
    "test = pd.read_csv(dpath +\"RentListingInquries_FE_test.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>virtual</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3758</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3300</td>\n",
       "      <td>1650.000000</td>\n",
       "      <td>1100.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4900</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 227 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.0         1   2950      1475.000000     1475.000000        0.0   \n",
       "1        1.0         2   2850      1425.000000      950.000000       -1.0   \n",
       "2        1.0         1   3758      1879.000000     1879.000000        0.0   \n",
       "3        1.0         2   3300      1650.000000     1100.000000       -1.0   \n",
       "4        2.0         2   4900      1633.333333     1633.333333        0.0   \n",
       "\n",
       "   room_num  Year  Month  Day  ...   virtual  walk  walls  war  washer  water  \\\n",
       "0       2.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "1       3.0  2016      6   24  ...         0     0      0    1       0      0   \n",
       "2       2.0  2016      6    3  ...         0     0      0    0       0      0   \n",
       "3       3.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "4       4.0  2016      4   12  ...         0     0      0    1       0      0   \n",
       "\n",
       "   wheelchair  wifi  windows  work  \n",
       "0           0     0        0     0  \n",
       "1           0     0        0     0  \n",
       "2           0     0        0     0  \n",
       "3           1     0        0     0  \n",
       "4           0     0        0     0  \n",
       "\n",
       "[5 rows x 227 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现测试集中没有y，这次作业无法得到在测试集上的表现情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********** train **********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49352 entries, 0 to 49351\n",
      "Columns: 228 entries, bathrooms to interest_level\n",
      "dtypes: float64(9), int64(219)\n",
      "memory usage: 85.8 MB\n",
      "************ test ***********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Columns: 227 entries, bathrooms to work\n",
      "dtypes: float64(9), int64(218)\n",
      "memory usage: 129.3 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"*********** train **********\")\n",
    "train.info()\n",
    "print(\"************ test ***********\")\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集中少一维，即y标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAELCAYAAAARNxsIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGx5JREFUeJzt3X2wHXWd5/H3h/AgPibIhcokYYKaGUUdo14hypSj6EDAHRMpUChHIkNtxAKF0rUEd5aMPMyM4ygrM8pupogkrkOIiBI1GDMMyII8JGB4CJHlioxcyZJgeNQVivDZP/p35eTm5N7OpU9OTu7nVdV1ur/96z7fwy340v379a9lm4iIiCbs0e0EIiJi95GiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIas2e3E9jZ9t9/f0+fPr3baURE9JTbbrvtEdt9o7Ubd0Vl+vTprFmzpttpRET0FEn/Uaddbn9FRERjOlZUJL1I0q2S7pC0TtLnS/xSSb+QtLYsM0tcki6SNCDpTklvaTnXPEn3lWVeS/ytku4qx1wkSZ36PRERMbpO3v56GjjC9lOS9gJukHR12fcZ21cMa380MKMshwEXA4dJ2g9YAPQDBm6TtNz2o6XNfOBmYAUwG7iaiIjoio5dqbjyVNncqywjzbM/B1hSjrsZmChpMnAUsMr25lJIVgGzy76X277J1fz9S4C5nfo9ERExuo72qUiaIGktsJGqMNxSdl1QbnFdKGmfEpsCPNhy+GCJjRQfbBNvl8d8SWskrdm0adML/l0REdFeR4uK7S22ZwJTgUMlvQE4G3gt8DZgP+CzpXm7/hCPId4uj4W2+2339/WNOiIuIiLGaKeM/rL9GHAdMNv2hnKL62ng68ChpdkgMK3lsKnAQ6PEp7aJR0REl3Ry9FefpIllfV/gvcDPSl8IZaTWXODucshy4KQyCmwW8LjtDcBK4EhJkyRNAo4EVpZ9T0qaVc51EnBVp35PRESMrpOjvyYDiyVNoCpey2x/X9K/S+qjun21Fji1tF8BHAMMAL8FTgawvVnSecDq0u5c25vL+seBS4F9qUZ9ZeRXREQXqRo4NX709/c7T9RH7NoO/6fDu53Cbu/GT9y4Q+0l3Wa7f7R2eaI+IiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMR0rKpJeJOlWSXdIWifp8yV+sKRbJN0n6XJJe5f4PmV7oOyf3nKus0v8XklHtcRnl9iApLM69VsiIqKeTl6pPA0cYftNwExgtqRZwBeAC23PAB4FTintTwEetf0a4MLSDkmHACcArwdmA1+TNEHSBOCrwNHAIcCJpW1ERHRJx4qKK0+Vzb3KYuAI4IoSXwzMLetzyjZl/3skqcSX2n7a9i+AAeDQsgzYvt/2M8DS0jYiIrqko30q5YpiLbARWAX8HHjM9rOlySAwpaxPAR4EKPsfB17ZGh92zPbiERHRJR0tKra32J4JTKW6snhdu2blU9vZt6PxbUiaL2mNpDWbNm0aPfGIiBiTnTL6y/ZjwHXALGCipD3LrqnAQ2V9EJgGUPa/AtjcGh92zPbi7b5/oe1+2/19fX1N/KSIiGijk6O/+iRNLOv7Au8F1gPXAseVZvOAq8r68rJN2f/vtl3iJ5TRYQcDM4BbgdXAjDKabG+qzvzlnfo9ERExuj1HbzJmk4HFZZTWHsAy29+XdA+wVNL5wE+BS0r7S4BvSBqgukI5AcD2OknLgHuAZ4HTbG8BkHQ6sBKYACyyva6DvyciIkbRsaJi+07gzW3i91P1rwyP/w44fjvnugC4oE18BbDiBScbERGNyBP1ERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGjMqEVF0ksk7VHW/0jS+yXt1fnUIiKi19S5UrkeeJGkKcA1wMnApZ1MKiIielOdoiLbvwWOBf7J9geo3gkfERGxlVpFRdLbgQ8DPyixTk6ZHxERPapOUTkTOBv4Tnm3yauoXrQVERGxlVGvOGz/GPixpJeU7fuBT3Y6sYiI6D11Rn+9vbytcX3ZfpOkr3U8s4iI6Dl1bn/9d+Ao4NcAtu8A3tnJpCIiojfVevjR9oPDQls6kEtERPS4OqO4HpT0DsCS9qbqT1nf2bQiIqIX1blSORU4DZgCDAIzy3ZERMRWRi0qth+x/WHbB9o+wPZf2v71aMdJmibpWknrJa2TdEaJ/42kX0laW5ZjWo45W9KApHslHdUSn11iA5LOaokfLOkWSfdJurxcSUVERJfUGf21WNLElu1JkhbVOPezwKdtvw6YBZwmaehJ/AttzyzLinLeQ4ATgNcDs4GvSZogaQLwVeBoqif5T2w5zxfKuWYAjwKn1MgrIiI6pM7trz+x/djQhu1HgTePdpDtDbZvL+tPUvXDTBnhkDnAUttP2/4FMAAcWpYB2/fbfgZYCsyRJOAI4Ipy/GJgbo3fExERHVKnqOwhadLQhqT92MFpWiRNpypEt5TQ6ZLulLSo5dxTgNZRZoMltr34K4HHbD87LB4REV1Sp6h8CfiJpPMknQf8BPiHul8g6aXAt4EzbT8BXAy8mqrDf0M5P4DaHO4xxNvlMF/SGklrNm3aVDf1iIjYQXU66pcAxwEPAxuBY21/o87Jy3tXvg180/aV5XwP295i+zngX6hub0F1pTGt5fCpwEMjxB8BJkrac1i83W9YaLvfdn9fX1+d1CMiYgzqvvnxZ8CVwFXAU5IOGu2A0udxCbDe9pdb4pNbmn0AuLusLwdOkLSPpIOBGcCtwGpgRhnptTdVZ/5y26aa2PK4cvy8kl9ERHTJqH0jkj4BLKC6UtlCddvJwJ+McujhwEeAuyStLbHPUY3emlnO8QDwMYAyA/Iy4B6qkWOn2d5ScjgdWAlMABbZXlfO91lgqaTzgZ9SFbGIiOiSOh3uZwB/XOfZlFa2b6B9v8eKEY65ALigTXxFu+PKjMmHDo9HRER31Ln99SDweKcTiYiI3lfnSuV+4DpJPwCeHgq29pNERERAvaLyy7LsXZaIiIi26rz58fMAkl5i+zedTykiInpV3vwYERGNyZsfIyKiMXnzY0RENCZvfoyIiMbkzY8REdGYEa9UyguyPmL7wzspn4iI6GEjXqmUubfm7KRcIiKix9XpU7lR0j8DlwO/f05l6K2OERERQ+oUlXeUz3NbYqZ6lW9ERMTvjdansgdwse1lOymfiIjoYaP1qTwHnL6TcomIiB5XZ0jxKkn/RdI0SfsNLR3PLCIiek6dPpW/Kp+tz6YYeFXz6URERC+rM0vxwTsjkYiI6H113lF/Uru47SXNpxMREb2szu2vt7Wsvwh4D3A7kKISERFbqXP76xOt25JeAXyjYxlFRETPqjX1/TC/BWaM1qiMFrtW0npJ6ySdUeL7SVol6b7yOanEJekiSQOS7pT0lpZzzSvt75M0ryX+Vkl3lWMukqQx/J6IiGhInTc/fk/S8rJ8H7gXuKrGuZ8FPm37dcAs4DRJhwBnAdfYngFcU7YBjqYqVjOA+cDF5fv3AxYAhwGHAguGClFpM7/luNk18oqIiA6p06fyjy3rzwL/YXtwtINsbwA2lPUnJa2nmj5/DvCu0mwxcB3w2RJfYtvAzZImSppc2q6yvRlA0ipgtqTrgJfbvqnElwBzgatr/KaIiOiAOkXll8AG278DkLSvpOm2H6j7JZKmA28GbgEOLAUH2xskHVCaTQFa3zA5WGIjxQfbxCMiokvq9Kl8C3iuZXtLidUi6aXAt4EzbT8xUtM2MY8h3i6H+ZLWSFqzadOm0VKOiIgxqlNU9rT9zNBGWd+7zskl7UVVUL5p+8oSfrjc1qJ8bizxQWBay+FTgYdGiU9tE9+G7YW2+2339/X11Uk9IiLGoE5R2STp/UMbkuYAj4x2UBmJdQmw3vaXW3YtB4ZGcM3j+U7/5cBJZRTYLODxcptsJXCkpEmlg/5IYGXZ96SkWeW7TqLeAIKIiOiQOn0qpwLfLC/qguoKoe1T9sMcDnwEuEvS2hL7HPD3wDJJp1D11xxf9q0AjgEGqIYtnwxge7Ok84DVpd25Q532wMeBS4F9qTro00kfEdFFdR5+/Dkwq/SNyPaTdU5s+wba93tA9VT+8PZm60krW/ctAha1ia8B3lAnn4iI6Lw6z6n8raSJtp8qQ4MnSTp/ZyQXERG9pU6fytG2HxvasP0o1W2qiIiIrdQpKhMk7TO0IWlfYJ8R2kdExDhVp6P+fwHXSPo61XMgf0X1JHxERMRW6nTU/4OkO4H3ltB5tld2Nq2IiOhFda5UAH4K7EV1pfLTzqUTERG9rM7orw8CtwLHAR8EbpF0XKcTi4iI3lPnSuW/Am+zvRFAUh/wb8AVnUwsIiJ6T53RX3sMFZTi1zWPi4iIcabOlcoPJa0ELivbH6KaUiUiImIrdUZ/fUbSscCfUk27stD2dzqeWURE9Jxao7/KtPVXjtowIiLGtfSNREREY1JUIiKiMdstKpKuKZ9f2HnpRERELxupT2WypD8D3i9pKcPejWL79o5mFhERPWekonIOcBbVu9+/PGyfgSM6lVRERPSm7RYV21cAV0j6b7bP24k5RUREj6rznMp5kt4PvLOErrP9/c6mFRERvajOhJJ/B5wB3FOWM0osIiJiK3UefnwfMNP2cwCSFlNNf392JxOLiIjeU/c5lYkt66/oRCIREdH76hSVvwN+KunScpVyG/C3ox0kaZGkjZLubon9jaRfSVpblmNa9p0taUDSvZKOaonPLrEBSWe1xA+WdIuk+yRdLmnvuj86IiI6Y9SiYvsyYBbV3F9XAm+3vbTGuS8FZreJX2h7ZllWAEg6BDgBeH055muSJkiaAHwVOBo4BDixtAX4QjnXDOBR4JQaOUVERAfVuv1le4Pt5bavsv1/ax5zPbC5Zh5zgKW2n7b9C2AAOLQsA7bvt/0MsBSYI0lUz8kMvShsMTC35ndFRESHdGPur9Ml3Vluj00qsSnAgy1tBktse/FXAo/ZfnZYvC1J8yWtkbRm06ZNTf2OiIgYZmcXlYuBVwMzgQ3Al0pcbdp6DPG2bC+03W+7v6+vb8cyjoiI2kYsKpL2aO1of6FsP2x7Sxme/C9Ut7egutKY1tJ0KvDQCPFHgImS9hwWj4iILhqxqJT/+N8h6aAmvkzS5JbNDwBDBWs5cIKkfSQdDMwAbgVWAzPKSK+9qTrzl9s2cC1wXDl+HnBVEzlGRMTY1Xn4cTKwTtKtwG+GgrbfP9JBki4D3gXsL2kQWAC8S9JMqltVDwAfK+daJ2kZ1RP7zwKn2d5SznM6sBKYACyyva58xWeBpZLOp3oY85I6PzgiIjqnTlH5/FhObPvENuHt/off9gXABW3iK4AVbeL38/zts4iI2AXUmVDyx5L+EJhh+98kvZjqqiEiImIrdSaU/M9Uz4P8zxKaAny3k0lFRERvqjOk+DTgcOAJANv3AQd0MqmIiOhNdYrK0+VpdgDKMN7tPhMSERHjV52i8mNJnwP2lfTnwLeA73U2rYiI6EV1ispZwCbgLqohwCuAv+5kUhER0ZvqjP56rkx5fwvVba97y8OHERERWxm1qEh6H/A/gJ9Tzbl1sKSP2b6608lFRERvqfPw45eAd9seAJD0auAHQIpKRERspU6fysahglLcD2zsUD4REdHDtnulIunYsrpO0gpgGVWfyvFUEz1GRERsZaTbX3/Rsv4w8GdlfRMwadvmEREx3m23qNg+eWcmEhERva/O6K+DgU8A01vbjzb1fUREjD91Rn99l2rK+u8Bz3U2nYiI6GV1isrvbF/U8UwiIqLn1SkqX5G0APgR8PRQ0PbtHcsqIiJ6Up2i8kbgI8ARPH/7y2U7Ypf0y3Pf2O0UxoWDzrmr2ynELqZOUfkA8KrW6e8jIiLaqfNE/R3AxE4nEhERva/OlcqBwM8krWbrPpUMKY6IiK3UKSoLxnJiSYuA/0Q1d9gbSmw/4HKqZ14eAD5o+1FJAr4CHAP8Fvjo0EAASfN4/v0t59teXOJvBS4F9qV6x8sZmZI/IqK7Rr39ZfvH7ZYa574UmD0sdhZwje0ZwDVlG+BoYEZZ5gMXw++L0ALgMOBQYIGkoSliLi5th44b/l0REbGTjVpUJD0p6Ymy/E7SFklPjHac7euBzcPCc4DFZX0xMLclvsSVm4GJkiYDRwGrbG+2/SiwCphd9r3c9k3l6mRJy7kiIqJL6rz58WWt25LmUl01jMWBtjeU826QdECJTwEebGk3WGIjxQfbxNuSNJ/qqoaDDjpojKlHRMRo6oz+2ort79L8Mypq91VjiLdle6Htftv9fX19Y0wxIiJGU2dCyWNbNvcA+hnhP+CjeFjS5HKVMpnnX/Y1CExraTcVeKjE3zUsfl2JT23TPiIiuqjOlcpftCxHAU9S9YGMxXJgXlmfB1zVEj9JlVnA4+U22UrgSEmTSgf9kcDKsu9JSbPKyLGTWs4VERFdUqdPZUzvVZF0GdVVxv6SBqlGcf09sEzSKcAvqd4iCdWQ4GOAAaohxSeX794s6Tyef9PkubaHOv8/zvNDiq8uS0REdNFIrxM+Z4TjbPu8kU5s+8Tt7HpPu5MBp23nPIuARW3ia4A3jJRDRETsXCNdqfymTewlwCnAK4ERi0pERIw/I71O+EtD65JeBpxBdVtqKfCl7R0XERHj14h9KuWJ9k8BH6Z6WPEt5SHEiIiIbYzUp/JF4FhgIfBG20/ttKwiIqInjTSk+NPAH1BN5vhQy1QtT9aZpiUiIsafkfpUdvhp+4iIGN9SOCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGdKWoSHpA0l2S1kpaU2L7SVol6b7yOanEJekiSQOS7pT0lpbzzCvt75M0rxu/JSIintfNK5V3255pu79snwVcY3sGcE3ZBjgamFGW+cDFUBUhYAFwGHAosGCoEEVERHfsSre/5gCLy/piYG5LfIkrNwMTJU0GjgJW2d5s+1FgFTB7ZycdERHP61ZRMfAjSbdJml9iB9reAFA+DyjxKcCDLccOltj24hER0SXbfUd9hx1u+yFJBwCrJP1shLZqE/MI8W1PUBWu+QAHHXTQjuYaERE1deVKxfZD5XMj8B2qPpGHy20tyufG0nwQmNZy+FTgoRHi7b5voe1+2/19fX1N/pSIiGix04uKpJdIetnQOnAkcDewHBgawTUPuKqsLwdOKqPAZgGPl9tjK4EjJU0qHfRHllhERHRJN25/HQh8R9LQ9/+r7R9KWg0sk3QK8Evg+NJ+BXAMMAD8FjgZwPZmSecBq0u7c21v3nk/IyIihtvpRcX2/cCb2sR/DbynTdzAads51yJgUdM5RkTE2OxKQ4ojIqLHpahERERjujWkuCe89TNLup3Cbu+2L57U7RQiokG5UomIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjer6oSJot6V5JA5LO6nY+ERHjWU8XFUkTgK8CRwOHACdKOqS7WUVEjF89XVSAQ4EB2/fbfgZYCszpck4REeNWrxeVKcCDLduDJRYREV2wZ7cTeIHUJuZtGknzgfll8ylJ93Y0q+7aH3ik20nUpX+c1+0UdiU99bcDYEG7fwXHrZ76++mTO/y3+8M6jXq9qAwC01q2pwIPDW9keyGwcGcl1U2S1tju73YesePyt+tt+ftVev3212pghqSDJe0NnAAs73JOERHjVk9fqdh+VtLpwEpgArDI9roupxURMW71dFEBsL0CWNHtPHYh4+I2324qf7velr8fIHubfu2IiIgx6fU+lYiI2IWkqOwmMl1N75K0SNJGSXd3O5fYMZKmSbpW0npJ6ySd0e2cui23v3YDZbqa/wP8OdUw69XAibbv6WpiUYukdwJPAUtsv6Hb+UR9kiYDk23fLullwG3A3PH8716uVHYPma6mh9m+Htjc7Txix9neYPv2sv4ksJ5xPqtHisruIdPVRHSZpOnAm4FbuptJd6Wo7B5qTVcTEZ0h6aXAt4EzbT/R7Xy6KUVl91BrupqIaJ6kvagKyjdtX9ntfLotRWX3kOlqIrpAkoBLgPW2v9ztfHYFKSq7AdvPAkPT1awHlmW6mt4h6TLgJuCPJQ1KOqXbOUVthwMfAY6QtLYsx3Q7qW7KkOKIiGhMrlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYkAJP2kRpszJb24w3nMHO05B0kflfTPDX9v4+eM8SlFJQKw/Y4azc4EdqiolNcS7IiZwLh+eC56W4pKBCDpqfL5LknXSbpC0s8kfVOVTwJ/AFwr6drS9khJN0m6XdK3yqSCSHpA0jmSbgCOl/RqST+UdJuk/y3ptaXd8ZLulnSHpOvLFDvnAh8qT2Z/qEbefZK+LWl1WQ6XtEfJYWJLuwFJB7Zr3/g/zBjX9ux2AhG7oDcDr6ealPNG4HDbF0n6FPBu249I2h/4a+C9tn8j6bPAp6iKAsDvbP8pgKRrgFNt3yfpMOBrwBHAOcBRtn8laaLtZySdA/TbPr1mrl8BLrR9g6SDgJW2XyfpKuADwNfLdz5g+2FJ/zq8PfC6F/jPK+L3UlQitnWr7UEASWuB6cANw9rMAg4BbqzmFGRvqvm7hlxejn8p8A7gW6UdwD7l80bgUknLgLHObvte4JCWc7+8vIHwcqqi9XWqCUYvH6V9RCNSVCK29XTL+hba/3siYJXtE7dzjt+Uzz2Ax2zPHN7A9qnlKuJ9wFpJ27SpYQ/g7bb/31bJSTcBr5HUB8wFzh+l/Ri+OmJb6VOJqO9JYOj/6m8GDpf0GgBJL5b0R8MPKC9s+oWk40s7SXpTWX+17VtsnwM8QvVOnNbvqONHVDNUU845s3yvge8AX6aalv3XI7WPaEqKSkR9C4GrJV1rexPwUeAySXdSFZnXbue4DwOnSLoDWAfMKfEvSrpL0t3A9cAdwLVUt6dqddQDnwT6Jd0p6R7g1JZ9lwN/yfO3vkZrH/GCZer7iIhoTK5UIiKiMemoj9hFSToZOGNY+Ebbp3Ujn4g6cvsrIiIak9tfERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN+f+UQGoL+CmGNwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.interest_level);\n",
    "pyplot.xlabel('interest_level');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总共有三类样本，每类样本分布的并不均匀，一般类别之间的数量差别在十倍以上时才考虑使用class_weight，这里暂不考虑"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['interest_level']\n",
    "\n",
    "train = train.drop([\"interest_level\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二轮参数调整得到的最优值max_depth = 5 ， min_child_weight = 2，本轮返回来调节n_estimators,除了需要调优的参数和已暂时确认的两个参数，其余参数继续默认值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再次调整弱学习器数目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#直接调用xgboost内嵌的交叉验证（cv），可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证\n",
    "def modelfit(alg, X_train, y_train, cv_folds=None, early_stopping_rounds=10):\n",
    "    xgb_param = alg.get_xgb_params()\n",
    "    xgb_param['num_class'] = 3\n",
    "    \n",
    "    #直接调用xgboost，而非sklarn的wrapper类\n",
    "    xgtrain = xgb.DMatrix(X_train, label = y_train)\n",
    "        \n",
    "    cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], folds =cv_folds,\n",
    "             metrics='mlogloss', early_stopping_rounds=early_stopping_rounds)\n",
    "  \n",
    "    cvresult.to_csv('2_nestimators.csv', index_label = 'n_estimators')\n",
    "    \n",
    "    #最佳参数n_estimators\n",
    "    n_estimators = cvresult.shape[0]\n",
    "    \n",
    "    # 采用交叉验证得到的最佳参数n_estimators，训练模型\n",
    "    alg.set_params(n_estimators = n_estimators)\n",
    "    alg.fit(X_train, y_train, eval_metric='mlogloss')\n",
    "    \n",
    "    #Predict training set:\n",
    "    train_predprob = alg.predict_proba(X_train)\n",
    "    logloss = log_loss(y_train, train_predprob)\n",
    "        \n",
    "    #Print model report:\n",
    "    print (\"logloss of train :\" )\n",
    "    print (logloss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of train :\n",
      "0.5646101854810868\n"
     ]
    }
   ],
   "source": [
    "xgb2_3 = XGBClassifier(\n",
    "        learning_rate =0.01,\n",
    "        n_estimators=1000,  #虽然知道n_estimator应该在337左右，但设置1000也没关系，会自动返回最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.5,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb2_3, X_train, y_train, cv_folds = kfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: FutureWarning: from_csv is deprecated. Please use read_csv(...) instead. Note that some of the default arguments are different, so please refer to the documentation for from_csv when changing your function calls\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8HPWd//HXZ7UqVrNsS65yAxuDDaYZ00soCRAChADBgVwgEFLgSM8PLrmEIyGQHCkkcAkkAS4kmKMFCJDQIVSDbUyzcQH3KhdZxZZkaT+/P2Ykr21VW6tZad/Px2Me2ik7+5kZe98732nm7oiIiADEoi5ARETSh0JBRERaKBRERKSFQkFERFooFEREpIVCQUREWigURJKY2X+Y2R+jrkMkKgqFXsbMCs1siZl9LmlYkZktM7Nzk4ZNMbPHzGyTmVWa2Vwzu97MBoTjLzazJjOrCbuPzOyrKa79BDNbkcrP6IrW6nH3n7r7ZSn6vCVmdnIq5p0KPbW9ett66esUCr2Mu9cAlwM3m1lZOPjnwEx3fwDAzI4CXgBeAfZ19xLgVKARODBpdq+5e6G7FwLnAj83s4N7ZkmkK8wsHnUNkiHcXV0v7IC7gOnACcAGYFjSuJeB33bw/ouBl3ca9gbwuaT+M4H3gUqCkNkvadx+4bDKcJozk8adDswFqoGVwHeAAmArkABqwm54G8t1K/B4+P4ZwN6dWB/7Ak8DG4H5wPm7Uw9wLfCX8H1jAAcuAZYDm4CvAIcB74TLfkvS5+wNPBduj/XAX4GScNzd4WdtDT/re51Yx0uA/xd+Vj0QD/tXhssyHziplXVxBLAGyEoa9mngnfD1VGAmUAWsBX7Zxjo9AVjRxrj+wJ+BCmAp8AMgFo7LAn4RroPFwJXheoy3Ma8lwMltjPsSsCjcro82/5sBDPgVsA7YHK6j/dva3lH/f+1NXeQFqNvNDQcDgNXhf7xLkoYXAE3ACR28/2KSQiH8oqsE9gn79wFqgVOAbOB74X/OnLB/EfAfYf+J4X/ACeF7VwPHJtV5SPi6zS+ZpDruCr8ApoZfgn8F7u3gPQUEX9qXhO85JFwvk7paD62Hwu+BPODjQB3wMDAYGBF+KR0fTj8uXF+5QBnwL+DXSfPe4cuvvXWcNP0cYCTQD5gQLufwpPpaDUzgQ+CUpP77gavD168Bnw9fFwJHtDGPNrcXQSA8AhSFdSwALg3HfYXgS7k8XN/PsBuhEP67Wh9uz1zgt8C/wnGfAGYBJQQBsR/hD6O2tre6znVqPuql3H0TwS/MfOChpFEDCJoF1zQPMLOfh8cVas3sB0nTHhEOryHYS7gbWBiO+yzwuLs/7e7bgJsIvpiOIvglWgjc6O4N7v4c8BgwLXzvNmCimRW7+yZ3n93FxXvI3d9w90aCUDiog+nPAJa4+53u3hh+3oMETWLdUc+P3b3O3Z8i+BKf7u7r3H0l8BJwMIC7LwrXV727VwC/BI5vZ77treNmv3H35e6+lSDsc8NlyXb3Je7+YRvznk64PcysiODX8/Sk9THOzErdvcbdX+/KyjCzrLD2a9y92t2XEOwZfD6c5HzgZndfEf47vbEr809yIXCHu89293rgGuBIMxsTLkMRwR6iufs8d1+dtHx7sr0zmkKhlzKziwh+oT0D/Cxp1CaCZophzQPc/XseHFf4G8Ev6Wavu3uJB8cUhgKTgJ+G44YTNAs0zyNB8Ct1RDhueTis2dJwHMBnCL6ElprZi2Z2ZBcXb03S6y0EAdSe0cDhYcBVmlklwRfK0G6qZ23S662t9BcCmNlgM7vXzFaaWRXwF6C0nfm2t46bLU8avwj4BsHezLrws4a3Me97gHPMLBc4B5jt7s2fdSnBXsoHZvammZ3RTo2tKSXYQ1yaNCx5+w9Prnun112x8/qpIWiaGxH+ELmFoKlxrZndbmbF4aR7ur0zmkKhFzKzwQTtqV8Cvgycb2bHAbh7LUE7/Dldmae7ryX4df2pcNAqgi/b5s80gmaMleG4kWaW/O9nVDgOd3/T3c8iaGJ5GLiv+WO6UlMXLAdeDAOuuSt096/2cD03hPOc7O7FwEUETRvNdv689tZxq+9x93vc/Zjwfc6OPwiSp5tL8IV6GvA5gpBoHrfQ3acRrI+fAQ+YWUHnF5P1BL/GRycNa9n+BM035UnjRnZh3sl2Xj8FwCC2/zv7jbsfSvBjZh/gu+Hwtra3dIJCoXe6BXjY3Z8Pd5m/B/wh/FVI2P9FM7s6DBDMrBwY29YMzWwQwcHI98NB9wGfNLOTzCwb+DbBwc5XCUKnFviemWWb2QkEYXKvmeWY2YVm1j9sEqkiaPaA4Bf2IDPr303rodljwD5m9vmwnmwzO8zM9uvheooIDiJXmtkIwi+pJGuBvZL621vHuzCzCWZ2Yrid6wj2UppamzZ0D3AVcBzBMYXm+VxkZmXhnkllOLjN+ZhZXnJHsCd6H3B9eDr0aOBbBHtGzcv1dTMbYWYlBAfHO5K90+fEw/ovMbODwmX+KTDD3ZeE2/fwcL3VhuujqYPtLZ0R9UENdV3rgLMJfkGV7DT8WeD6pP7DgScI/tNXAu8B1wODwvEXE/xnaT7zZh1Bm/PgpHl8muCA4WbgRcIDt+G4SeGwzeE0nw6H5wD/JGjGqgLeBI5Jet8dBE0AlbR99tFPkvpPoIOD0+F0EwjOWKoI5/8cwbGILtVD6wea40nTryDpID7BF+EPktbJrHB9ziH4kl+RNO1ZwLLws77TiXW8hB0PTE8mOPZTTXAw/rHW1mHS9KMIvsAf32n4X8LtXUPwI+DsNt5/Qrj8O3fjCI5d/SVc38uBH7L97KM4wZ7sBoKzj75JsGdhbXzOklY+4yfhuK8QHDRvXt7ycPhJBGcc1bD9TK/Cjra3uo47C1ewiEhKmNlpwO/dfXSHE0vk1HwkIt3KzPqZ2elmFg+b0X5EcJKD9ALaU5BewcyOBf7R2jgPzp6SNGFm+QRNYfsSHPd4HPi6u1dFWph0ikJBRERaqPlIRERa9LqbbJWWlvqYMWOiLkNEpFeZNWvWencv62i6XhcKY8aMYebMmVGXISLSq5jZ0o6nUvORiIgkUSiIiEiLlIWCmd1hZuvM7L02xu9rZq+ZWb2ZfSdVdYiISOelck/hLoKnfbVlI8F9WW5KYQ0iItIFKQsFd/8XwRd/W+PXufubBPdEERGRNKBjCiIi0qJXhIKZXW5mM81sZkVFRdTliIj0Wb0iFNz9dnef4u5Tyso6vPaiVSsXz+e1+37BlprN3VydiEjf0StCoTusm/8qR869jlUftnoylIiIkMIrms1sOsFDOkrNbAXB7XOzAdz992Y2FJgJFAMJM/sGMDFVd1IsKd8PgMoVH8CBR6fiI0REer2UhYIHz4Btb/wadnyOa0oNGzsRgG3rFvTUR4qI9DoZ03yUV1DMOgYSr1wcdSkiImkrY0IBYH1uOUW1y6IuQ0QkbWVUKNQWjmZw4wr0YCERkdZlVCgkBoxjINVUbtS1DiIircmoUOg3dDwAaxbrtFQRkdZkVCgMGBmcllq1UmcgiYi0JqNCYeiYfUm40VixMOpSRETSUkaFQnZuPmtjZeRUfhR1KSIiaSmjQgFgQ245/bcuj7oMEZG0lHGhsLVoDEMbV+KJRNSliIiknYwLBQbuTbFtYd3aVVFXIiKSdjIuFPKHTQBgnU5LFRHZRcaFQtnYSQBUr5wXcSUiIukn80KhfB/qPRuvmB91KSIiaSfjQsGy4qzKLid/84dRlyIiknYyLhQANhfsRVn9kqjLEBFJOxkZCo0DxzPcK9hUWRl1KSIiaSUjQyFv+ERi5qxc9G7UpYiIpJWMDIXSsZMB2LxMoSAikiwjQ2Hw6Ik0eoz5782MuhQRkbSSkaEQy85lTXw4++eujboUEZG0kpGhALApfyxlWxdHXYaISFrJ2FDYNnA85b6GzTW1UZciIpI2MjYUcofuR7Y1seLD96MuRUQkbWRsKAwcE5yBtGmJzkASEWmWsaEwZK8DSLjRtHZu1KWIiKSNjA2FWG4Bq7OG0W/TB1GXIiKSNjI2FADWF4xnyFbdGE9EpFnKQsHM7jCzdWbW6tNsLPAbM1tkZu+Y2SGpqqUtjWUTGelrqNi4sac/WkQkLaVyT+Eu4NR2xp8GjA+7y4HfpbCWVhWMnEzMnBUfzO7pjxYRSUspCwV3/xfQ3k/ws4A/e+B1oMTMhqWqntYMnzAFgKqlc3ryY0VE0laUxxRGAMuT+leEw3ZhZpeb2Uwzm1lRUdFtBRQPHUctebBO1yqIiEC0oWCtDPPWJnT32919irtPKSsr674KYjFW5+7FgCo9mlNEBKINhRXAyKT+cmBVTxfxbmM5oxsXU9fQ2NMfLSKSdqIMhUeBfwvPQjoC2Ozuq3u6iPEHHEF/28KSjxb29EeLiKSdVJ6SOh14DZhgZivM7FIz+4qZfSWc5AngI2AR8Afga6mqpT2D9joYgIq//zCKjxcRSSvxVM3Y3ad1MN6BK1L1+Z01ZFxwecS7daUcG3EtIiJRy+grmgFi+SWsjQ3h4OzlHU8sItLHZXwoAKwv2o/yrQtIJFo9+UlEJGMoFICmYQcy0tayZMXKqEsREYmUQgEYsPfhAKya91rElYiIREuhAAzf7wgAti6dFXElIiLRUigAWYWDWJc1hPwNegqbiGQ2hUJoQ/FERtYtoLEpEXUpIiKRUSiEfNiBjLJ1LFq2IupSREQio1AIDRwXHGxePe/1iCsREYmOQiE0ZEIQCnXLdLBZRDKXQiFkBcHB5sKNrT49VEQkIygUkrzvYxlVt4C6bU1RlyIiEgmFQpLhE49idGwdCxYviboUEZFIKBSSlO53HABr3n854kpERKKhUEgyaPzhbPMsVrz9fNSliIhEQqGQLCef1f3GMdkWEDzuQUQksygUdlKbyGZiYhHLK6qiLkVEpMcpFHZSdOxXybd6Fr2ni9hEJPMoFHYybP/gYPPsl5+MuBIRkZ6Xsmc091ZZJSPZmFXKIbGFUZciItLjtKewMzM22AD2aZjL5i3boq5GRKRHKRRakX3g+ZTbet79YG7UpYiI9CiFQiuG7n88AOvn6SI2EcksCoVW5I08mK2eQ82Cl6IuRUSkRykUWhPPYU3/yRzKXLY0NEZdjYhIj1EotMEa65nAMubMXxx1KSIiPUah0Iah59xAzJzV7zwXdSkiIj1GodCGvDFTaSCH+PJXoi5FRKTHpDQUzOxUM5tvZovM7OpWxo82s2fN7B0ze8HMylNZT5fEc1nTfzLjtszR9QoikjFSFgpmlgXcCpwGTASmmdnEnSa7Cfizu08GrgNuSFU9uyM25hj2s2Vc/Lunoi5FRKRHpHJPYSqwyN0/cvcG4F7grJ2mmQg8G75+vpXxkRoy+WRi5pxf92DUpYiI9IhUhsIIYHlS/4pwWLK3gc+Erz8NFJnZoJ1nZGaXm9lMM5tZUVGRkmJbkz16Kg2WQ56p+UhEMkMqQ8FaGbbzk2u+AxxvZm8BxwMrgV0uDHD32919irtPKSsr6/5K2xLPZW3JIUyqm0VFdX3Pfa6ISERSGQorgJFJ/eXAquQJ3H2Vu5/j7gcD3w+HbU5hTV0W3+ck9omtZOY770ZdiohIyqUyFN4ExpvZWDPLAS4AHk2ewMxKzay5hmuAO1JYz24ZctDpAGx8V89XEJG+L2Wh4O6NwJXAk8A84D53f9/MrjOzM8PJTgDmm9kCYAhwfarq2V2xoZPYHB/EoDUv05TQc5tFpG9L6UN23P0J4Imdhv0w6fUDwAOprGGPmVE1/FgOX/oMc5au59CxPXhMQ0Skh+mK5k4YNPlUBlgN82brVtoi0rcpFDohf79TAGh4696IKxERSa0OQ8HM9jaz3PD1CWZ2lZmVpL60NFJQyiovZVJsCeuq6qKuRkQkZTqzp/Ag0GRm44A/AWOBe1JaVRrKLhjAIbaQS297JupSRERSpjOhkAjPJPo08Gt3/yYwLLVlpZ/SC24h25r4eJVueSEifVdnQmGbmU0DvgA8Fg7LTl1J6cnKD6Pa+zGGVdRta4q6HBGRlOhMKFwCHAlc7+6LzWws8JfUlpWGYlmstVKOj73DSx+s6nh6EZFeqMNQcPe57n6Vu083swFAkbvf2AO1pZ0xpUUU2xYe/Nv9UZciIpISnTn76AUzKzazgQR3Nb3TzH6Z+tLST/zyZ9jmWRzd8AqNTYmoyxER6XadaT7q7+5VwDnAne5+KHByastKUzkFrLIyjo+9zRsfbYi6GhGRbteZUIib2TDgfLYfaM5YQwcUMypWwVuzXou6FBGRbteZULiO4KZ2H7r7m2a2F7AwtWWlr9xLghu9Zr3/gG6QJyJ9TmcONN/v7pPd/ath/0fu/pmO3tdnFQ9jY3wwJ9hsZqgJSUT6mM4caC43s7+Z2TozW2tmD5pZeU8Ul66KCovZN7ac12e8EnUpIiLdqjPNR3cSPBxnOMEzlv8eDstY2Zc+TgKj38K/U9+oC9lEpO/oTCiUufud7t4YdncBmf1QgaKhbB48lVMSr/Cpm1+KuhoRkW7TmVBYb2YXmVlW2F0EZHxjevGh5zEutooBtRl7zF1E+qDOhMIXCU5HXQOsBs4luPVFRsuadDYJYpzQ+ApVdduiLkdEpFt05uyjZe5+pruXuftgdz+b4EK2zFZYRu3woziNV3n8bd0LSUT6ht198tq3urWKXqqwbi1jYmt557Unoy5FRKRb7G4oWLdW0UvZl5+nweMcsP4JFq2rjrocEZE9truhoEt5AXKLSEw6hzOyXueyP+osJBHp/doMBTOrNrOqVrpqgmsWBMibchHFtpWjG2ewTXdOFZFers1QcPcidy9upSty93hPFpnWxhxLXSyfj297jtN+/a+oqxER2SO723wkzWIxcgoHcUzsXax6ZdTViIjsEYVCN4hd8jgxgzMan+aDNVVRlyMistsUCt1hwGga8wZyQdbzXPLHV6OuRkRkt6U0FMzsVDObb2aLzOzqVsaPMrPnzewtM3vHzE5PZT2plF00hCFWydSGGbrCWUR6rc7cOru1s5CWh7fT3qud92UBtwKnAROBaWY2cafJfgDc5+4HAxcA/7P7ixKxr75MQ8FwzvWneGDmiqirERHZLZ3ZU/gl8F2C22aXA98B/gDcC9zRzvumAovCh/I0hNOftdM0DhSHr/sDvfd+EbEscuJxjs16j+n/eF6np4pIr9SZUDjV3W9z92p3r3L324HT3f3/gAHtvG8EsDypf0U4LNm1wEVmtgJ4Avj31mZkZpeb2Uwzm1lRUdGJkiNy2dMkMKbZkzz2Tu/NNxHJXJ0JhYSZnW9msbA7P2lce1c2t3YrjJ2nnwbc5e7lwOnA3Wa2S03ufru7T3H3KWVlafwoh6Kh2AHn8dms57nxoddx14XfItK7dCYULgQ+D6wLu88T/LrvB1zZzvtWACOT+svZtXnoUuA+AHd/DcgDSjtVeZqyo/6dAqvn04mnOVUXs4lIL9OZW2d/5O6fcvfSsPuUuy9y963u/nI7b30TGG9mY80sh+BA8qM7TbMMOAnAzPYjCIU0bh/qhGGTSYw5nkviT7Juc03U1YiIdElnzj4qD880Wmdma83sQTMr7+h97t5IsCfxJDCP4Cyj983sOjM7M5zs28CXzOxtYDpwsfeBNpfY0VcxxDbxsYZ/MXPJxqjLERHpNOvoO9jMngbuAe4OB10EXOjup6S4tlZNmTLFZ86cGcVHd547iVuPYNH6rfxo2G1M//KRUVckIhnOzGa5+5SOpuvMMYUyd7/T3RvD7i4gjY/2pgEzYt7EPiwld+mzvLpofdQViYh0SmdCYb2ZXWRmWWF3EbAh1YX1el99Fe8/km/GH+Lyu9/UmUgi0it0JhS+CJwPrAFWA+cCl6SyqD4hnoNl5XJg7EOmbJvN8/PXRV2RiEiHOnP20TJ3P9Pdy9x9sLufDZzTA7X1fl97Dc/K5bvx/+PKe2aTSGhvQUTS2+7eEO9b3VpFXxXPwfqPZFJsKVMbZ/PAbN0TSUTS2+6GQmtXK0trwr2Fa+LT+f7f3qGmvjHqikRE2rS7oaB2kM6K52AlI5kQW87JPoOP//LFqCsSEWlTm6HQxi2zq8ysGhjegzX2fle8Adn5XJv9Z9ZX1bBkfW3UFYmItKrNUHD3IncvbqUrcvd4TxbZ68WyYMAYhtgmvhD7J2fd+rJOURWRtKTHcfaUr74Kef35ds5D+NZKTlEzkoikIYVCTzGDix8n1+u4Lvt/WbpxC5u36rGdIpJeFAo9aegBWNEwzsp6hX0SiznpFy9EXZGIyA4UCj3ta69jsWz+J+c3bKip48n310RdkYhIC4VCT+tXAgPGMNrWcGHOy1zx19msr6mPuioREUChEI0r3oDcYv4r608MTGzkiJ8+q7ORRCQtKBSiEIvB5S+QRRP3Fd5EYyLBSb/Q2UgiEj2FQlQG7Q0loxjTuJgr4o+yZEMtC9ZWR12ViGQ4hUKU/n02WBbfjt/PIN/Mqb/+l+6NJCKRUihEKZYFX3uNGM6TxT8m4c5xP39exxdEJDIKhaiVTYABYxjYsIqril5gY20Dx/38+airEpEMpVBIB1e9BbE439z2B/bPWsbyTVt54t3VUVclIhlIoZAOzODb87GsHB7tdx1FtpWv/XU2by3bFHVlIpJhFArpoqAULnqQWOMWZuZdQW7cOO/3r7Fsw5aoKxORDKJQSCdjj4MBY8n1Ol49+m0ATvnVi6zZXBdxYSKSKRQK6SY8vjBoxo08dUY99Y0JjrrxWTbWNkRdmYhkAIVCujGDq5eDxdjrqYt59NxiEg5TfvI0lVsUDCKSWgqFdJSTD994DzAmP/Yppp83jITDIT9+WjfPE5GUUiikq/4j4GuvAc6Rj53E9IvGk3CYev0zOsYgIimT0lAws1PNbL6ZLTKzq1sZ/yszmxN2C8ysMpX19DqD94OLnwBPcOSDR3D/pQeTcDjyxmdZvL426upEpA9KWSiYWRZwK3AaMBGYZmYTk6dx92+6+0HufhDwW+ChVNXTa405Gs77X/AmDrtnfx758qG4w8dueoEZH22IujoR6WNSuacwFVjk7h+5ewNwL3BWO9NPA6ansJ7ea9LZcOZvwZs48O5JvPitI4kZfPb21znxpheirk5E+pBUhsIIYHlS/4pw2C7MbDQwFniujfGXm9lMM5tZUVHR7YX2Cof8GwzcGxKNjL7zEN665jiyYsZH62s57CfPULetKeoKRaQPSGUoWCvD2rr95wXAA+7e6jebu9/u7lPcfUpZWVm3FdjrXDU7CIatm+j/q9Es+M+jGd4/j4qaeib+8J+6+llE9lgqQ2EFMDKpvxxY1ca0F6Cmo865ajac8wfwJrJ+PoZXrzqQP31hCgmH4/77eZ6ZuzbqCkWkF0tlKLwJjDezsWaWQ/DF/+jOE5nZBGAA8FoKa+lbJp8Pn7sPPAH/vTcnlVby0vc+BsBlf57J1Ouf0cN6RGS3pCwU3L0RuBJ4EpgH3Ofu75vZdWZ2ZtKk04B7XU+W6Zp9PgGXPhO8vnUqIze+xgc/PpVh/fNYV13P/j96klc/XB9tjSLS61hv+y6eMmWKz5w5M+oy0kflcrh5crDXMGAv+PpbzFyykXN/H+x4DSnO5ZlvHU9RXnbEhYpIlMxslrtP6Wg6XdHc25WMDO6VFIvDpo/gpn2YMjyPededypDiXNZW1XPAtU/xyJyVesyniHRIodAX5BbCD9ZB8UioWQs3jKDf5kXM+I+TefiKowH4+r1zGPf9fzBvdVXExYpIOlMo9BWxLPjWe3DRQ0FT0q1TYc49HDSyhA9/ejpjBuXTlHBOu/klxv3HEyzfqNNXRWRXOqbQF1Wthl9NAm+CgjK48k3oN4DKLQ2cdvNLrA5vqDekOJdHrjiGof3zIi5YRFKts8cUFAp9VaIJbj4INi8L+i+YDvueDsDqzVs587evUBHehntwUS73f+VIRg8qiKpaEUkxhYIEVs2B248PXheUwRVvQP5AAJZt2MJJv3yBbU3Bv4FBBTncfenhTBxeHFW1IpIiCgXZrrEBfnsIbA5vRfWp38DBn4dYcEhpbVUdZ93yCmuqtj+n4TfTDubUSUPJieuwk0hfoFCQXa15D247LjjWAMHFbyMPaxlduaWBM295mWUbt7YMG94/j79cdjh7lRX2dLUi0o0UCtI6d3j3AXjosqA/vxQuexoG7tUySSLhvLiggsv+PJOmRPDvI2bwk7MP4JOTh9G/ny6EE+ltFArSvvpqePUWePHGoH/ql+GYb0LxsB0mW1tVx8NvreSGf3zQMmxgQQ7Xn70/J0wYTL+crJ6sWkR2k0JBOqd6Ddx2PNSsCfqLhsGXnoPi4TtM5u68u3IzX/rfmaytrm8ZHo8Zvzj/QE7cd7BupSGSxhQK0jWblsCfPh5cEQ1w2GVw9DeC22jspLEpwRuLN/KFO99oOXOp2Xc/MYHj9ylj4rBiYrHWHqkhIlFQKMju2bQ0DIdwzyEWhy8+BeWHtjp5IuHMXraJq6a/xarNdTuMKy3M4YefmsSx40oZUJCT6spFpB0KBdkzlcuD01ibGoJ+y4Jz/wT7ngFZbTcTrauu46UF6/nO/W/v8pi9Lx07lsPHDuKwMQPpn6+mJpGepFCQ7lFfDW/9Ff75/7YPO+Zb4TOjx7b71qaE886KSl6YX8HNzy7cZfyQ4ly+/8mJHFjen1ED8zFTc5NIqigUpHslmmDRM3Dv5yARPtXNsuC8u2DCae3uPTSr29bEnOWVvLF4I798esEu47NixpeP24vJ5SUcOLI/Q4vzFBQi3UShIKmzeSW8dTe8cMP2YYd9CSZ9GkYdEdyxtRO2NSWYv6aad1du5oePvLfLQetmV500nglDipgwtJDRgwrIztJV1iJdpVCQ1GtqhEVPwyNXwJYN24dPvRwmnt2lgGhWt62JeaureGfFZn782FwaE63/+xxUkMMXjxnL+MGFTBhaxMgB+TrbSaQdCgXpWfU1sPBJeOjy7c1LEFwUN+lsGHlEy72WuqpuWxMfVtSwYG01P//n/JZbf7emtDCHy4/bi/FDitirtIARJf2Ia89CRKEgEaqvhgVPwoOX7jj88K8EexAjp3Z5D6I1NfVvg/jHAAAQA0lEQVSNLFxbzYK11fzg4babn5pdfNQYxgzKZ3RpAWMHFTBiQD81RUnGUChIemgOiL99ecc9iPxSOPla2PtE6D+iWz9yY20DC9dWs3TDFpZsqOV/Xviww/fEY8YXjxnLsP55DC/px4iSfgwv6ceA/Gwd7JY+QaEg6ae+GhY+Bf+4GmrX7TjuyCth74/B6KMhu19KPt7dWV/TwNINtSxeX8vKyq3c/MzCXa6naEtZYQ4XHjGa4SX9GN6/H8NLggDJy9b9nyT9KRQkvbnDurmw6Fl4+j93HGdZcOIPggPVww+B7J55XKi7s7G2gdWb61hZuZVVlVv56RPzOmyWSjakOJeLjxrLoMIcSgtzGFSQy6Dwr24eKFFSKEjv0lALS16Bv18F1at3HJdbBIdeHBysHnUEFJRGUiJAQ2OCtVXbQ2NV5VZ+8dSCTu9tJCsrzOW8KeUMKsyltDCHgQVBeJQW5jCgIEfHO6RbKRSkd6tdD8tnwLLX4dXf7DjOYnDg54KAGHUEDBoHadTuv6WhkQ01DayvqWdjbQMbahr47XMLWb5pa8dvbkd2lvG1E8aFAZJLSX42xXnZ9O8XdIV5cbJ0Wq60QaEgfcu2Olj1Fix/HZ65dtfxEz4Jow4P9iaGHwTx3B4vcXe4O1VbG1lfW8+GmgY21NSzvraB37/wISsr9yxEkg0tzuPCw0fRPylIivvFw7/BMB0b6dsUCtK3JRKwYSEsew2eu37XA9cQXEQ3/OCgK92nW06DjVpjU4KNWxrYWNvA5i3b2Lw16KrqGrnj5Y9YWdn2NRx7woCrT9uXorxsivLiFObFKc6Lb+/PjVOQE9cFhGksLULBzE4FbgaygD+6+42tTHM+cC3gwNvu/rn25qlQkDbVrAuam5bPgNdu2XW8xeDgz8PQA2DI/jBkEuQV93ydEalvbKK6rnF7kCT9/fHj82hoTPRIHeUl/fjiMWMpzI3TLyeL/Jys8G88eJ0dDMvPiZOXHdMpwd0k8lAwsyxgAXAKsAJ4E5jm7nOTphkP3Aec6O6bzGywu7fyk287hYJ0WqIJNiwKmp1Wvw0zbgNv2nW6fc/YHhRDD4CSUWl1jCIdJBLOlm1NVNdto7quMey2v66p38YNT3ywWwfcu1N5ST8uPnrM9oDJaQ6YLPplx7e/DkMnk47BpEMoHAlc6+6fCPuvAXD3G5Km+TmwwN3/2Nn5KhRkj7hD1SpY8y6sfRde/z1sWb/rdJYVnPE0eD8o2zfoCst6vNy+qKExQW19I7UNjWxtaGJL2G3d1tjy+j8ffo/6Htpz6aryAf24+Kgx5GZnkZsVIzc7Rm48Rm48K/ibnfQ6nrXL+Kia2NIhFM4FTnX3y8L+zwOHu/uVSdM8TLA3cTRBE9O17v7P9uarUJCUaKiFtXODoFjzLsz639b3KiwLplwShETpPlA2AQqHaM8iTTQ0JoKgCQNme+hsD6DfPreQJRu2RF3qbpkwpJAnv3n8br23s6EQ3625d7KGVobtnEBxYDxwAlAOvGRm+7t75Q4zMrscuBxg1KhR3V+pSE4BjDws6ADO+FWwV1G9GtbNg4oPgr9z7oE3W9mxtSw48ILtQTFofNAMFddjSHtSTjxGTjxGf9p+vsdnDi3f7fm7Ow1NCeobE9RvS1Df2LTr68YE9duSXjc2cd3f53bLnk91XWPHE+2hqJuPfg+87u53hf3PAle7+5ttzVd7ChK55rBYvwAqFsD6+TDzztb3LABy+8PET8GAsTBgTPB34FjoN0B7GNJj0qH5KE7QNHQSsJLgQPPn3P39pGlOJTj4/AUzKwXeAg5y9w2tzRMUCpLmtlbC+oXB6bKblsKmJfDOvW1Pn10A405MCoywKxnVqafZiXRW5M1H7t5oZlcCTxIcL7jD3d83s+uAme7+aDju42Y2F2gCvtteIIikvX4lOzZDAZxzW/C3oXZ7UDR3b9wG8/7exswMxh67fe8iOTTyB6ZuGSSj6eI1kXSQSEDNGti4eHtgzLoTaivafk9OAYw7eXtolIyC4hFQPDyjrr+Qzom8+ShVFAqSkeproHLpjqHx7gNQt6nt98T7Bbf+KBoeBMUO3QjIH6RjGhkk8uYjEelGuYXBFdhDJm0f9smbgr+JpuDAd+Wy4BqMqlVBf9VKWPQcNLzQ9nxLRod7F8O2h0XRsO17HIVDIEtfE5lEW1ukt4tlQf/yoGtLoim4DUjVKqgOg6NqJVSthvcebPvMKQhDItzDKEra02gOkqLhPfbMC0k9hYJIJohlhV/iw4BDdxz3mT8Ef91h66YwLFbt2H3wGKyc1f5nDDkgDIykPY3m0CgaCnn91VzVCygURCRgFpzVlD8wuAfUDm7d/rK+OtjDaA6P6qTwWNDuDQkCIw8PmqWKhoZ/h0HRECgcGgzrNxBiesBQVBQKItI1uUVQVgRl+7Q9TWN9eFwj6RhH9RqoWRv8/eAx8A6u8B1+cBAaLd3goCsYvP11TqH2PrqZQkFEul88d/s1Fe1p2BIERXNYtLxeG5yiu/gl2Fbb8eeNmBIGRmkQGgVlwQ0MC8q2h0heifZAOkGhICLRyckPbvkxcGz70zU1wpYNwcOUasKu+XVtRfD3oxdgWydudGcxGHsc5JeGoTEo+JtfGoZKWXC6boYeA1EoiEj6y4oHxx2KhnQ8bSIRHDBPDo3mrmZd8PzvhU+1f8ZVsuwCGDk1CIzm4MgftGvXb0CfOH239y+BiEiyWCz89T8oeB5GR7bVBc/UqF2//W/L6wqo3dC1ELEYjDg0KTAGth4i+YPSsklLoSAimS07r+PrPJI11sOWjUFz1g7dTsMWPtXxwfRmg8Z3IkQGBgfccwt3f1k7QaEgItIV8dykaz46qWFL+wGyZQNs3QgL/tF+kBSPgG/NbXt8N1AoiIikWk5+0JWM7Nz07tBQs2uIlI5PbZ0oFERE0o9ZcD1IblHHp/V2s/Q6wiEiIpFSKIiISAuFgoiItFAoiIhIC4WCiIi0UCiIiEgLhYKIiLRQKIiISAtz96hr6BIzqwCW7ubbS4H13VhOb6Blzgxa5sywJ8s82t3LOpqo14XCnjCzme4+Jeo6epKWOTNomTNDTyyzmo9ERKSFQkFERFpkWijcHnUBEdAyZwYtc2ZI+TJn1DEFERFpX6btKYiISDsUCiIi0iJjQsHMTjWz+Wa2yMyujrqe7mJmI83seTObZ2bvm9nXw+EDzexpM1sY/h0QDjcz+024Ht4xs0OiXYLdY2ZZZvaWmT0W9o81sxnh8v6fmeWEw3PD/kXh+DFR1r0nzKzEzB4wsw/C7X1kX97OZvbN8N/0e2Y23czy+uJ2NrM7zGydmb2XNKzL29XMvhBOv9DMvrC79WREKJhZFnArcBowEZhmZhOjrarbNALfdvf9gCOAK8Jluxp41t3HA8+G/RCsg/Fhdznwu54vuVt8HZiX1P8z4Ffh8m4CLg2HXwpscvdxwK/C6Xqrm4F/uvu+wIEEy98nt7OZjQCuAqa4+/5AFnABfXM73wWcutOwLm1XMxsI/Ag4HJgK/Kg5SLrM3ft8BxwJPJnUfw1wTdR1pWhZHwFOAeYDw8Jhw4D54evbgGlJ07dM11s6oDz8j3Ii8BhgBFd5xnfe3sCTwJHh63g4nUW9DLuxzMXA4p1r76vbGRgBLAcGhtvtMeATfXU7A2OA93Z3uwLTgNuShu8wXVe6jNhTYPs/sGYrwmF9SrjLfDAwAxji7qsBwr+Dw8n6wrr4NfA9IBH2DwIq3b0x7E9eppblDcdvDqfvbfYCKoA7w2azP5pZAX10O7v7SuAmYBmwmmC7zaLvb+dmXd2u3ba9MyUUrJVhfepcXDMrBB4EvuHuVe1N2sqwXrMuzOwMYJ27z0oe3Mqk3olxvUkcOAT4nbsfDNSyvUmhNb16ucOmj7OAscBwoICg6WRnfW07d6St5ey25c+UUFgBjEzqLwdWRVRLtzOzbIJA+Ku7PxQOXmtmw8Lxw4B14fDevi6OBs40syXAvQRNSL8GSswsHk6TvEwtyxuO7w9s7MmCu8kKYIW7zwj7HyAIib66nU8GFrt7hbtvAx4CjqLvb+dmXd2u3ba9MyUU3gTGh2cu5BAcsHo04pq6hZkZ8Cdgnrv/MmnUo0DzGQhfIDjW0Dz838KzGI4ANjfvpvYG7n6Nu5e7+xiC7ficu18IPA+cG0628/I2r4dzw+l73S9Id18DLDezCeGgk4C59NHtTNBsdISZ5Yf/xpuXt09v5yRd3a5PAh83swHhXtbHw2FdF/UBlh48kHM6sAD4EPh+1PV043IdQ7Cb+A4wJ+xOJ2hPfRZYGP4dGE5vBGdifQi8S3B2R+TLsZvLfgLwWPh6L+ANYBFwP5AbDs8L+xeF4/eKuu49WN6DgJnhtn4YGNCXtzPwX8AHwHvA3UBuX9zOwHSC4ybbCH7xX7o72xX4Yrj8i4BLdrce3eZCRERaZErzkYiIdIJCQUREWigURESkhUJBRERaKBRERKSFQkFERFooFEQ6wcwOMrPTk/rPtG66BbuZfcPM8rtjXiJ7StcpiHSCmV1McKHQlSmY95Jw3uu78J4sd2/q7lpEtKcgfYqZjQkfQPOH8AEtT5lZvzam3dvM/mlms8zsJTPbNxx+Xvhgl7fN7F/hrVGuAz5rZnPM7LNmdrGZ3RJOf5eZ/c6Chx19ZGbHhw9OmWdmdyV93u/MbGZY13+Fw64iuOHb82b2fDhsmpm9G9bws6T315jZdWY2AzjSzG40s7nhw1ZuSs0alYwT9SXe6tR1Z0dwX/pG4KCw/z7gojamfRYYH74+nOB+ORDcPmBE+Lok/HsxcEvSe1v6CR6Sci/BLQjOAqqAAwh+dM1KqqX5VgVZwAvA5LB/CVAavh5OcN+fMoI7oz4HnB2Oc+D85nkR3EvfkutUp25PO+0pSF+02N3nhK9nEQTFDsJbjR8F3G9mcwgeSjIsHP0KcJeZfYngC7wz/u7uThAoa939XXdPAO8nff75ZjYbeAuYRPAUwJ0dBrzgwd1BG4G/AseF45oI7oYLQfDUAX80s3OALZ2sU6Rd8Y4nEel16pNeNwGtNR/FCB7YctDOI9z9K2Z2OPBJYI6Z7TJNO5+Z2OnzE0DczMYC3wEOc/dNYbNSXivzae2++M3qPDyO4O6NZjaV4O6hFwBXEtxGXGSPaE9BMpIHDyJabGbnQcsD0Q8MX+/t7jPc/YcEj3UcCVQDRXvwkcUED8bZbGZD2PGBMcnzngEcb2al4bPFpwEv7jyzcE+nv7s/AXyD4A6qIntMewqSyS4EfmdmPwCyCY4LvA38t5mNJ/jV/mw4bBlwddjUdENXP8jd3zaztwiakz4iaKJqdjvwDzNb7e4fM7NrCJ4bYMAT7v7IrnOkCHjEzPLC6b7Z1ZpEWqNTUkVEpIWaj0REpIWaj6TPM7NbCZ7tnOxmd78zinpE0pmaj0REpIWaj0REpIVCQUREWigURESkhUJBRERa/H+tdf0UB0wY4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cvresult = pd.DataFrame.from_csv('2_nestimators.csv')\n",
    "        \n",
    "# plot\n",
    "test_means = cvresult['test-mlogloss-mean']\n",
    "test_stds = cvresult['test-mlogloss-std'] \n",
    "        \n",
    "train_means = cvresult['train-mlogloss-mean']\n",
    "train_stds = cvresult['train-mlogloss-std'] \n",
    "\n",
    "x_axis = range(0, cvresult.shape[0])\n",
    "        \n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'n_estimators5_1.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: FutureWarning: from_csv is deprecated. Please use read_csv(...) instead. Note that some of the default arguments are different, so please refer to the documentation for from_csv when changing your function calls\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAANGCAYAAAAcVgSJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4Xnd9///nW5Jv25L3tuMVx85OyF4ESIEAZZRCB6O0zJaWlhJoy49RaFil7bdsSillpNBSoNAAYQYCoSELsvfw3rK8ZEuyLEv6/P44R7KiyLYkyz73eD6u61w697nPfe73PexLL31WpJSQJEmSJA1PXdEFSJIkSVIlMURJkiRJ0ggYoiRJkiRpBAxRkiRJkjQChihJkiRJGgFDlCRJkiSNgCFKkiRJkkbAECVJkiRJI2CIkiRJkqQRMERJqgoR8cWI2B8RZw1x3zsiIkXEiwYdn5Lfd3tE7I6IAxHRHBE/iohXRsT4Aecuza8xcNsTEfdGxFURUX88XufhRMSbIuI1RddxNCKiMSKujogrhrjvNfn7vrSAuhbkdZ1zvJ+7SBFxTUS0FV0HQP7+p4iYVXQtkmSIklQtrgK2Av8REeP6Duah6n3ANSml6wYcXwHcDbwb+CXwR8AzgTcDm4AvAn87xPN8Crg0334fuBn4GPBPY/+SRuxNwGuKLuIoNQJ/B1wxxH3fJ3vftxzPgnILyOqqqRAlSRpaQ9EFSNJYSCntiYjXA9eThZ+/y8PUV4BmspAFQEQ0AN8GZgAXpZQeHnS5b0TE+4Fzh3iq9Sml2wbc/lFEnAm8AvirMXtBepKUUgvQUnQdYykiGlNKHUXXIUkaGVuiJFWNlNJPgc8C74qI84GrgacAr08ptQ449SXA6cCHhghQfddal1L69jCfuhU4MPBARNRFxNsj4pG8m+G2iPhyRCwc/OCIeF3eLbAzInZGxLURcdqgc5ZFxNciYnN+veaIuKGve1lErAXOAJ4xoLvh2sMVnZ/z6Yj4w4h4OCI68jpeOMzXPfBapYj42wGvtyUivhQRswed98yIuDEidkTEvohYHxHfyrvxLeVgSPq7Aa/jmvyxT+rOl1/rgYi4NCJuya+5NiJem9//goi4K39t90fE8wbVszyv8/H8nE0Rcd3AbqF518Jf5ze/NKCuqwec81sRcWt+jb0R8ZOIuHTQc/V1RzsvIr4ZEbuAVfl9h/18D/GeX5Vfb/kQ9/1jRHT1dX2LiHMj4nv593B//jzfH+r7OBoRcXle7978PbglIl5wiPNuzb/rmyLiAxHxhsGf61HWcmZEfCciduXPc09EvHrQOXX59/XR/DuzOyLui4i3DDhndkR8LiI2DPhO3xwRzx6LOiVVNluiJFWbvwGeC3wTWAR8NqX0k0HnXJn//O4orl8XWUsWwFTgxcDzgH8cdN6/An8CfBr4HrAU+ABwRUScl1LaDhAR7wT+Hvhv4J3ATLLwd2tEXJhSejy/3g+AeuDtwHpgFnAZMC2//yX5a24l69YHsH8Yr+cFwIXAe4G2/PrXRsQpKaXVw3g8EVEHfAd4Glm3xluAJWTdKG+MiAtSSvvyX5K/D9wEvA7YDZxA9v6VyLrpPQ/4EfAF4PP5Uxyp9Wke8KX8uTeSdcn8YkQsAn6X7P1tzV/jtyNiWUppc/7YBcAO4B3588wAXg3cHhHnppQeBe4CXps/xwfz10D+XETEK4H/ImsFfQUwPn8fb4yIZ6WUfjmo3v8FvkYW+JvyY0f6fIfyn2Tfu9cwoOtpZOPzXgVcl1LaHhFNwE+ANcCfk7XMzgN+A5h8mOsPS0Q8I7/+fcDryb53bwKui4hXpJS+np93dn7eY2TvcQfwp3mtYyIiTiH7/m0D/pLss30VcE1EzE0p9XW7fTvZv7MPAv8HjANO5Ynv91eA88i6/D6W33ce2b9RSbUupeTm5uZWVRvZL7KJ7JfySUPc/8P8/vGDjgfZH5f6tvoB9y3NHzPU9qVB556aH/+XQde/KD/+ofz2NLJfJL8/6LxFQCfwX/ntmfnj3nKE1/0AcOMI3qdENo5s8oBjc4Ee4B0juM7L82u9dNDxC/Ljf5bf/p389lMOc61Z+TlXD3Hfa/L7lg44dmN+7PwBx2YA3fl7u2DA8afk5775MM9fT/YL9WPAR4d4La8ZdH4d2Ri6+4C6AccnkYWVmwccuzq/xvsGXWNYn+8h6v0WsGHQc/9mfr0X5rfPz2+/eBTXvwZoO8I5t+avddKAY/XA/XltkR/7BllQnzXo/Xtw8Od6iOfpe/9mHeac/87/7SwadPwHQDswNb99HXD3EZ5vL/Cxkb5nbm5utbHZnU9SVclbRd4M9AJzyH5xHq63kHXL69vuHeKcT5C13FxI9pf8d5FNMPHfA875jfznNQMfmFL6FfAw8Kz80KXAxCHO2wD8bMB5O8m6ff1NRLwt75o1Vv9//zyltHfAczeT/RV/yQiu8UKyVqXrIqKhbwPuIQtpV+Tn3QN0AZ+LiFdHxLKxeAHAlpTSnX03Uko7yV7DPelgixNk7z0MeG15re+KiIcioossfHUBK4AndKk8hFPIWrO+klLqHVBDG1nAuSQiGgc95luDbh/N5/slYCEwsIvZa8ne9x/mt1cCu4B/jIg/jYjTh3ntI8pbuS4Gvpm/ZgBSSj1kLTkLyd4jgGcAP0t5K2x+Xi9ZuBorzwRuyP8NDXQN2aQlfV0sfwU8JSI+ExHPjYgpQ1zrV8Br8m5/l8SACWskyRAlqdr8NdkvSq8EHifr1jVx0Dnr85+Dg8JXORiQ7jrE9TemlO7ItxtTSh8m66b3exHx3Pycvu4+Q80it3nA/cM6L6WUyALVj8m6Id0FtETEJyPiaLtj7Rji2H6ycDdcc8la1bp4Ygg9QNZtbBZASmkV2S/724B/AVZFxKqB41BGaecQx7oGH08pdeW7EwYc/ijZ5/dt4EVkgeBCsgA9nPfgSJ9hHTB90PEnnHuUn+8P8+v1jQGbDvwW8OU8yJCy8YDPIAuxfw88mI+Jet8YBIPpZC24h3r98MTve/MQ5w11bLRmDrOWD5P9X3EJ2Xu4Ix/TdcGAx7wM+A/gDWStbTsjG9c4bwzrlVShDFGSqkb+F/b3k/0C+XWy7l/LgQ8NOrVvjNRvDTyYUtrWF5DIuvIM1335z75Wr75gMn+IcxcA20d4Himb6OL1KaV5ZH/Z/xjZuJP/N4I6j5XtZK/lwkNsfWO0SCndlFJ6Edl4skvIfjn9eES8/HgXnXsV2fflXSmlH6eUfpV//sNdi+hIn2EvWSvQQGnwiaP9fAe0+Px2REwj++PBeLIWqoHn3Z9SejlZiDgH+DrZGLGjnVFyF9lrPNTrhyd+3+cOcd5YhpIdw6klpdSdUvpoSuk8su6fryDrRvvjvpbDlNL2lNJVKaWlZH9weSfwUga1HEuqTYYoSVUh7z72H2S/JL0FIGVTkX8UeEtEPHXA6dcCD5HN4nfqGDx93wxq2/KfP8t/PmHAfERcSNZF7Ib80K3AviHOW0jeLWmoJ0spPZZS+iDZmJPzBtw10haksfI9sl/O6we00g3cHh38gJRST0rpdrKJDuDg6+ibDON4vY7EoAk48lnlThh03qHqepRsTNQrIyIGXKOJbAzYrWmEU5gf5vM9lC+Rta69guwPB7emlB45xLVTSunelNJbybpgDuf6h6u1HbgdeOnAFt+8O+KryCbfeCw//AvgmTFgsdz8vN87mhoGuSF/jgWDjv8R2Ri52wY/IKW0O6X0TbLW0Rlk4x8Hn7M+pfRpsj/AHNV7Jqk6ODufpGrxTrLB/7+ZUto94Ph7yLppfTEizkkp7Usp9UTEb5N1n/pVRPw72QQFu8i6pV1M1qo01PTniyPikny/iazr4DuBdWSzrpFSejQiPge8OSJ6yboLLSXrNraBrJWBlNLuiPgA8PcR8WWycVUzyRZ17SSb3a5vVrNPA/9D1kWxiyxknQ38w4Da7gdeHhEvA1YDnSml+0fyJo7S14A/AH4QEZ8gG0tygGw8zG8A30kpXRsRf5rX/X2yLpUTyGbpA/gpQEppb0SsA14cETeQdcnbnlJae4xq/x7ZuJdHyFoUzyeb4XHjoPNWkQXeP4iIh8kmSNicUtocEW8nm53vexHxb2QtQX9D9l16x5EKGMHnO6SU0iMRcSvZ93AR2ayQA6//QrJWrW+TfS+CrEVlGgdbZQ+nPiJ+d4jj7SmlH+bP+xPg5xHxz3n9bwLOBF6Rd1eErEX4RcANEfEhsvfzTzk4Q2Evw/OiiHhSS3EehN5HNkbv55Gt9baT7Lv5AuDteddGIuI6solY7iCblXEJ2Vpy64DHI2Iq8HOyLr6PkLVMX0g2e+T/DrNOSdWs6Jkt3Nzc3I52Iws8XcDnDnH/JWQzzn100PEpZL8A/oqDaz01k01V/SagccC5S3nyrHz7yFoiPgbMG3TtOrLxLY/mtbWQD7Qfor7Xk43B2U/WOvBt4PQB988ha23o++V9b37+VTxxVsAlZMFwT17f2iO8bwn49BDH1wLXjPAzaCDrGnZP/r7szev9LLB8wOfwv/n1O8laDW8EXjToWs8iGxfUmdd4TX78NQw9O98Dh3gN3zvSayYLEp/PP/d2sunXL8+ve+Ogx748f01dDJpBkGyq+9vy195GFgovG/T4qxlidrnhfr5HeP//OL92BzBl0H2nkIWBlfn9u8laj149jOtew6FnpVw74LzLyVqB2vLnuJV8dsBB17s8f586ycYu/RPZv5NEPnPeYWq5+jC1pAHnnUm2fMFusn9T9/DkWRXfBtxM9u9yP1l4+jywJL9/PNkyBfeS/d/QQRamrmbA/wtubm61u/VNOypJknTcRcT1ZMH45KJrkaThsjufJEk6LiLio8DdZN1aZ5B1tbuSrDVWkiqGIUqSdEj5hB2H05sGrI8kHUE92Qya88i64T0E/GFK6T8LrUqSRsjufJKkIUXEUmDNEU57X0rp6mNejCRJZcSWKEnSoWwmm5HsSOdIklRTbImSJEmSpBFwsV1JkiRJGoGa686Xryi/gGwdDkmSJEm1bTLZAurD7qJXcyGKLEANXolekiRJUu1aCGwa7sm1GKL2AmzYsIEpU6YUXYskSZKkguzZs4dFixbBCHup1WKIAmDKlCmGKEmSJEkj5sQSkiRJkjQChihJkiRJGgFDlCRJkiSNgCFKkiRJkkbAECVJkiRJI2CIkiRJkqQRMERJkiRJ0ggYoiRJkiRpBAxRkiRJkjQChihJkiRJGgFDlCRJkiSNgCFKkiRJkkbAECVJkiRJI2CIkiRJkqQRMERJkiRJ0ggYoiRJkiRpBAxRkiRJkjQChihJkiRJGgFDlCRJkiSNgCFKkiRJkkbAECVJkiRJI2CIkiRJkqQRMERJkiRJ0ggYoiRJkiRpBAxRkiRJkjQChihJkiRJGgFDlCRJkiSNgCFKkiRJkkbAECVJkiRJI2CIkiRJkqQRMEQV6Lp7N3PV1+7me/dtLroUSZIkScNkiCrQPRt28e17NvMXX72bjq7uosuRJEmSNAwNRRdQy5ZOqWdRNHPZKYtoLPlRSJIkSZXAlqgCPWPtR7lp/Fs5b9u3ii5FkiRJ0jAZogo0btpCABo7mwuuRJIkSdJwGaIK1DR7MQBTu1vo7uktuBpJkiRJw2GIKlDT7CUAzGMnLW37C65GkiRJ0nAYogpUP3UBAPNiJ1taOwuuRpIkSdJwGKIK1DFhDgBTYh+btm4ruBpJkiRJw2GIKlDj5Onsq2sCoHPnxoKrkSRJkjQchqiCtZVmA9C1c0PBlUiSJEkaDkNUwboa5wHQ07q54EokSZIkDYchqmC9k7PJJRrathRciSRJkqThMEQVrGF6tuDuRBfclSRJkiqCIapgDfk055O7trG380DB1UiSJEk6EkNUwWbMPxGAebGLfV09BVcjSZIk6UgMUQWrn3YC4IK7kiRJUqUwRBVtwjQAZsUetm5zwV1JkiSp3BmiijZ1IQdiHAB7dhmiJEmSpHJniCpaBHtLcwDo3OGCu5IkSVK5M0SVgc6J+YK7uzcVXIkkSZKkIzFElYHupixEbVq/io6u7oKrkSRJknQ4hqgyMC5fcHfFxD00lhoKrkaSJEnS4RiiysDEWYsAmNK1jd7eVHA1kiRJkg7HEFUGJs1eAsBcdrKzo6vgaiRJkiQdjiGqDDRMzRbcnRs72bLbBXclSZKkcmaIKgdTFgAwh91s3bW34GIkSZIkHY4hqgx0lGbQnepoiF62bXWtKEmSJKmcGaLKQOOE8bSXZgFwwLWiJEmSpLJmiCoTnRPnAnBg58aCK5EkSZJ0OIaoctDVTm9rFp7q9hiiJEmSpHJmiCoHpSYaznwpAOP3bSu4GEmSJEmHY4gqExNnLwVg2oFmurp7iy1GkiRJ0iEZospEw4xFACyI7azd0VZwNZIkSZIOxRBVLqYuBGBB7GCzC+5KkiRJZcsQVSYmzFwKZAvu7txjS5QkSZJUrgxR5aJpFgeiRF0k9mxbX3Q1kiRJkg7BEFUuImgbPw+A/dvXFVyMJEmSpEMxRJWRrkkLAEitrhUlSZIklStDVDnJJ5cY17ap4EIkSZIkHYohqozUT8umOZ/YsYX2/QcKrkaSJEnSUAxRZWTq/BOBbJrzzgMuuCtJkiSVI0NUGRk3fTGQLbjrWlGSJElSeTJElZPGmUDWErWpZWfBxUiSJEkaiiGqnMxaAcCk6KRl956Ci5EkSZI0FENUORk3kfaG6QB0bFtbbC2SJEmShmSIKjMdE7MFd3917710dHUXXI0kSZKkwQxRZaZ3SrZW1AXTO2gsNRRcjSRJkqTBDFFlpmF631pRmwuuRJIkSdJQDFFlpnHGAgBmdG9jX5uTS0iSJEnlxhBVZibMzWboWxA72LzPj0eSJEkqN/6WXmZiatadL1twd1/B1UiSJEkazBBVZjoas9n55rKLtdt2F1yNJEmSpMEMUWWmcdp8umMc9ZHo3LGp6HIkSZIkDWKIKjd1dbSPnwtA5451BRcjSZIkaTBDVLnpaqdr395sf/eGYmuRJEmS9CSGqHJTaiKWPzvb7dhScDGSJEmSBjNElaHSrMUATO7cSk9vKrgaSZIkSQMZosrQuOnZNOfz2M76ne0FVyNJkiRpIENUGZo4aymQLbi7s/1AscVIkiRJegJDVDlywV1JkiSpbBmiytHE6QBMiX20bHWtKEmSJKmcGKLK0aTZdDRMBWDfLkOUJEmSVE4MUWWqs3E+AN07XStKkiRJKieGqDJ1YNIJALRsWkVHV3fB1UiSJEnqY4gqU+NnLgFgScNOJo6rL7gaSZIkSX0MUWWqaU4Woub0trCzvavgaiRJkiT1MUSVqXGT5wJwQmxnw7adBVcjSZIkqY8hqlzNWgHkIWpvKrgYSZIkSX0MUWWqoymbWGIuu1jXbEuUJEmSVC4MUeWqcRYdaTx1kWhvWVt0NZIkSZJyhqgy1Th+HAcmL8xu7F5XbDGSJEmS+hmiyljv1EUA1Le64K4kSZJULgxR5aqrnXEbbwNg0r5N9PQ6uYQkSZJUDgxR5arURONz3g3AAlpo3tNZcEGSJEmSwBBV1vrGRC2KFlZu21twNZIkSZLAEFXWxs9aBsDCaGHb3q6Cq5EkSZIEhqjyNm0xALOjlS0trhUlSZIklQNDVDmbOJ399U0AtLesLrgYSZIkSWCIKm8HOujszmblSzvWFFyMJEmSJDBElbdSE2nJUwFoaNtUcDGSJEmSwBBV9upnLAFgSudmdnc4uYQkSZJUNENUmZs09yQgm6FvV8eBgquRJEmSZIgqczE9a4laFC1s2NlRcDWSJEmSDFHlbloWohZGCxt2GaIkSZKkohmiyl3TLABmRBvNW7cUXIwkSZIkQ1S5mzyPzoapAFx/2910dHUXXJAkSZJU2wxRFWD/pIUAXD6rncZSQ8HVSJIkSbXNEFUB+iaXKLVtKLgSSZIkSYaoCjBh9okAzDzQTNt+u/NJkiRJRTJEVYDStAUALIptbGjeUXA1kiRJUm0zRFWCWScDsDC2s2FvKrgYSZIkqbYZoirAvqasJWphbGP1tr0FVyNJkiTVNkNUBZg4exkAU2Ifu3dtL7gaSZIkqbYZoipBqZF9pRkAdG1fU3AxkiRJUm0zRFWIA5MXZTu71xdbiCRJklTjDFGVoKuduu2PAjChbQMpObmEJEmSVBRDVCUoNdFwyZ8AMKd3G5t27yu4IEmSJKl2GaIqRN+Cu4uihW179xdcjSRJklS7DFGVYtoSABZGC+t2tBdcjCRJklS7DFGVIg9Ri6KFtS2GKEmSJKkohqhK0TiDlKAx9rOz2Rn6JEmSpKIYoipF4ww6G+cB0LXTECVJkiQVxRBVQbqnZl36OptX0tHVXXA1kiRJUm0yRFWQCXOWA7A0mjnQ41pRkiRJUhEMURVk3KxlACyua2b9jo6Cq5EkSZJqkyGqkszIQtSS2MZapzmXJEmSCmGIqiST5wOwJJpZv213wcVIkiRJtckQVUnmnAbA7GjlX2542MklJEmSpAIYoirJxGl0jZsKwPNP6KSx1FBwQZIkSVLtKTxERcSbImJNRHRGxJ0R8bTDnDsuIt4bEavy8++NiOcdz3qL1j1tKQB1u9cUW4gkSZJUowoNURHxMuDjwIeAc4GbgB9GxOJDPOSDwBuBNwOnA58Fro2Ic49DuWWhYdZJAEzr3Gh3PkmSJKkARbdEvQ34Qkrp8ymlh1NKVwEbgD87xPl/CPx9SukHKaXVKaV/BX4M/NVxqrdwpRlZvlwS21i3dUfB1UiSJEm1p7AQFREl4Hzg+kF3XQ9cdoiHjQc6Bx3bB1x+mOcZHxFT+jZg8ihLLgv7p2ctUYujmcd22hIlSZIkHW9FtkTNAuqB5kHHm4F5h3jMj4G3RcSKiKiLiCuBFwPzD/M87wRaB2wbj6rqgo2fvRzIpjnf2jo4T0qSJEk61oruzgeQBt2OIY71eQvwOPAI0AV8GvgS0HOY638YmDpgW3g0xRZu+okAnBDb2dDSWnAxkiRJUu0pMkRtJws/g1ud5vDk1ikAUkotKaXfBpqAJcCpQBtwyKnqUkr7U0p7+jZg71gUX5jJ8+iun0B9JDpanKFPkiRJOt4KC1EppS7gTuDKQXddCdxyhMd2ppQ2AQ3A7wDfOSZFlqMDHRzo7s32d64uthZJkiSpBhW9WutHga9ExB3ArcCfAIvJpi4nIr4MbEopvTO/fTFwAnBP/vNqsiD4T8e98gLVkYWopo4N7Grfz/Sm8QVXJEmSJNWOQsdEpZS+DlwFvJcsGD0deH5KaV1+ymKeOGnEBLK1oh4CrgU2AZenlHYft6KLVmqidOkbgWya8x3tXQUXJEmSJNWWoluiSCl9BvjMIe67YtDtX5AtslvTYkY2ucSSaGbdjg6Wz6noWdslSZKkilIOs/NppGYsA7IQtXZHR8HFSJIkSbXFEFWJ8paoxbGN9dsre7JBSZIkqdIYoirRxBn0JpgQB7j+trvo6OouuiJJkiSpZhiiKtHEaXRNXgzABdPaaCwVPrRNkiRJqhmGqArVN7lEY9s6DvT0FlyNJEmSVDsMURWqNGc5AItoZvPufQVXI0mSJNUOQ1SFGjjNuTP0SZIkScePIapC7Z84F8hm6Hts0/aCq5EkSZJqhyGqQo1fcAaQtURt2uuYKEmSJOl4MURVqulLAZgW7TQ3bym2FkmSJKmGGKIqVamJrolzAOjevqrgYiRJkqTaYYiqYH2TS0xoW0/ngZ6Cq5EkSZJqgyGqUnW107DxdgCW0MyjW/cUXJAkSZJUGwxRlarURDzrPQCcWLeVLa2dBRckSZIk1QZDVCWbmS24e2JsYVVLe8HFSJIkSbXBEFXJ+kPUVtZsN0RJkiRJx4MhqpLNWAbA9GhjW/PmgouRJEmSaoMhqqIlulI9AG2bHqGjq7vgeiRJkqTqZ4iqZKUm6pdeBmRd+vYf6C24IEmSJKn6GaIqXP3sFQCcWLeF1Y6LkiRJko45Q1SlGzBD3+qWtoKLkSRJkqqfIarS5SFqmTP0SZIkSceFIarSTVkAwNLYyr/d6OQSkiRJ0rHWUHQBOkqzT6U3GphIFxfP7qGx5EcqSZIkHUu2RFW6+nH0TF0CQMPu1fT0poILkiRJkqqbIaoKNMzOxkUt6t3M5t37Cq5GkiRJqm6GqCrQPX0ZkM3Q9/CWPQVXI0mSJFU3Q1QVSFMWAtmCu2uadxVcjSRJklTdDFFVoLTgLCBridq0t6fgaiRJkqTqZoiqBvlaUYtjG+u2tRZcjCRJklTdDFHVYPJ8ehom0hC97G9ZXXQ1kiRJUlUzRFWD7n2kA/sBaGxby462/QUXJEmSJFUvQ1SVaIheIBsXtWZHe8HVSJIkSdXLEFUNSk3wtL8GYFlsZdMu14qSJEmSjhVDVLWYeRIAS2MrK7e1FVyMJEmSVL0MUdUin6HvxLothihJkiTpGDJEVYvJ8wFYEDv5xQOr6ejqLrggSZIkqToZoqrFtEX0TJgOwIr6bZTq/WglSZKkY8HftKtI3exTAViSNrNuZ0fB1UiSJEnVyRBVRWL2CgBOqtvkuChJkiTpGDFEVZGu6dnkEifFZh7esqfgaiRJkqTqZIiqIr1TFwNZiFrdvLvgaiRJkqTqZIiqIhMWngNkC+6u3+mCu5IkSdKxYIiqJtMW01s/nvFxgM6WNaSUiq5IkiRJqjqGqGpSV0/ki+7O797IltbOgguSJEmSqo8hqsr0zMxn6IvNPLi5teBqJEmSpOpjiKomXe00PPxtAJbHJjY4LkqSJEkac4aoalJqgt/5AgAn1W1mZYtrRUmSJEljzRBVbWadDGTd+VY2G6IkSZKksWaIqjb5xBIzoo2Va9fS0dVdcEGSJElSdTFEVZ1Ebz6z+UmxmV3tXcWWI0mSJFUZQ1QVqovs50l1m9m022nOJUmSpLFkiKo2pSa45E1APi5qm+OiJEmSpLFkiKpGs7K1opbHJkOUJEmSNMYMUdVo1ilA1hL1xZvXOLmEJEmSNIYMUdVo6kIAFsZ2lkxONJYaCi5IkiRJqh6GqGo0bTFpwjTqItHYtp7WjgNFVyRJkiRVDUNUNYogZh/s0vfYtr0FFyRJkiRVD0NUleqekS0zVHvPAAAgAElEQVS6e1Js5v5NrQVXI0mSJFUPQ1SV6p26BIDldZtYuWVXwdVIkiRJ1cMQVaVK804FsmnOV2/fV3A1kiRJUvUwRFWr+ecAsCy2sLJ5DymlgguSJEmSqoMhqlpNXUQa18j46GZq5wY27OoouiJJkiSpKhiiqlX3PuJAFpxWxCYeb24ruCBJkiSpOhiiasDJsZF1O2yJkiRJksaCIapalZrgyg8AcHLdRh5rdq0oSZIkaSwYoqrZnNMAWBEb+dqvN9DR1V1wQZIkSVLlM0RVs9nZNOcnxhYa6HaGPkmSJGkMGKKq2cTppASl6GFpbGX3PluiJEmSpKNliKpm4ycRC88HssklHtvquChJkiTpaBmiqt3sbFzUyXUbedTJJSRJkqSjZoiqdnOycVErYiP/8MNHnFxCkiRJOkqGqGo3/UQATo5NnDF/Eo2lhoILkiRJkiqbIarazT4ZgKWxlbUtrfT0OkOfJEmSdDQMUdVu5grS+CmMix5O6NnMuh3tRVckSZIkVTRDVLWLoHdW1hp1cmzkvo27Cy5IkiRJqmyGqBpQP2sFACvqNrJ6qyFKkiRJOhqGqFow+xQga4l6qLmt4GIkSZKkymaIqgXzzgayEPXw1o6Ci5EkSZIqmyGqFszJFtxdEs207N7DltZ9BRckSZIkVS5DVC2YPB/GT6UhelkWW1i3w9YoSZIkabQMUbXgQAfsbwXglFjPQ5v3FFyQJEmSVLkMUbWg1AQXvA6AU+s28PAWQ5QkSZI0WoaoWjH3DABOjfX8z50b6ejqLrggSZIkqTIZomrF3DOBrCWqVF/HuHo/ekmSJGk0/E26VkxfCsD82EljTyurWlwvSpIkSRoNQ1StmDwPpi0BHBclSZIkHQ1DVA3pnnM6kI2Lum9ja8HVSJIkSZXJEFVDGuadBWQh6vFmu/NJkiRJo2GIqiUzlwNwat16Ht7SSkqp4IIkSZKkymOIqiUnnA/AKbGRXe37WbejveCCJEmSpMpjiKolM04kNUxkYnSxJJp5ZOveoiuSJEmSKo4hqpbU1RNzTgOycVGrWmyJkiRJkkbKEFVLutph811A37gopzmXJEmSRsoQVUtKTfC8fwTgtFjP9+7bQkdXd8FFSZIkSZXFEFVr5p4BZN35APZ19RRZjSRJklRxDFG1Jg9Ri+tamEQHG3btK7ggSZIkqbIYompNw/j+3VNiAw9tdlyUJEmSNBKGqFpTaoIVzwHgtLr1vOva+x0XJUmSJI2AIaoWDRoXJUmSJGn4DFG1aO6ZAJxat4EJ4+oo1fs1kCRJkobL355r0YxlAJwW69h/oJvV2110V5IkSRouQ1Qtmnc2qWECTbGfJdHMnet2Fl2RJEmSVDEMUbWovoHe2acBcEasc4Y+SZIkaQQMUTWqPp9c4oy6tTy6pbXgaiRJkqTKYYiqVfOyySVOj3U8uKWN3t5UcEGSJElSZTBE1aqFFwJZS1R7Vw8Pb7VLnyRJkjQchqhaNed0iDpmRyuz2cXKbW1FVyRJkiRVBENUzUqQeoGsNeqBTY6LkiRJkobDECXOiHX8+01r6OjqLroUSZIkqewZompVqQmu/ACQtUQBpOTkEpIkSdKRGKJq2fyzATgzD1Hb27oKLEaSJEmqDIaoWjZzOQCLYxuT6eCBTc7QJ0mSJB2JIaqWTZzev3t6rOPPv3qX46IkSZKkIzBE1bJSE5zyAgBOr1vLZSfNpLHUUHBRkiRJUnkzRNW6fFzUGXXruH9jK729Ti4hSZIkHY4hqtbNOhmAM2MNe/d389AW14uSJEmSDscQVevmnQnA8thEiQNOLiFJkiQdgSGq1s1cAROn0xC9nBwbeGTr3qIrkiRJksqaIarWRcDcM4BsXNQ963cUXJAkSZJU3gxRgnlnAdm4qHs27mF3h4vuSpIkSYdiiBIsvBCAs+rWAMHKbW3F1iNJkiSVMUOUYMG5AJxet55xdPNos+OiJEmSpEMxRAma5gBQ4gAnx0befe0DdHR1F1yUJEmSVJ4MUcoml8idVbe6wEIkSZKk8meIEpSa4PK3AnB2rKYujnC+JEmSVMMMUcrk46LOG7eW3gQPbnbRXUmSJGkohihlZp8KwPLetYyni1+t2VlwQZIkSVJ5MkQpM+UEABqil1NiAw9ubi24IEmSJKk8GaKUGT8Jlj8bgLPrVnPXut0FFyRJkiSVJ0OUDpp7JgBnxyq27ulkdYuL7kqSJEmDGaJ00PyzAbiw7jEA7tlga5QkSZI0mCFKBy2+NPtR18IE9jtDnyRJkjQEQ5QOmjwfmuZQTw+nxzq+8Ms1dHR1F12VJEmSVFYMUTroQAe0bwOyySUAurp7i6xIkiRJKjuGKA3ponFrAFi9vb3gSiRJkqTyYojSQaUmeNW3ALigYRUAd63bVWRFkiRJUtkxROmJZp8GwJzuzUxnDx/8/sOOi5IkSZIGMETpiSZO6999St0qFkydQGOpocCCJEmSpPJiiNITlZrgKa8A4Ly6VWxu7WRra2fBRUmSJEnlwxClJ5t3FgBPrbsfgNtWby+yGkmSJKmsGKL0ZCecB8DpsY6gl3s2tBZckCRJklQ+DFF6snxyiYnRxYmxlfs3GaIkSZKkPoYoPdnEabDoEgDOjZXcuW4Xuzu6Ci5KkiRJKg+GKA1t/lMAuLjuIQAe2bqnyGokSZKksmGI0tDycVEX1T0KwEOb9xZZjSRJklQ2DFEa2pKnArCofjsT2M/7v/eQi+5KkiRJGKJ0KBOnA1Cfejgz1hRcjCRJklQ+DFEaWkT/7rn1qwBo67QlSpIkSTJEaWilJnjW3wHwlFgJwK2rdxRZkSRJklQWDFE6tLlnAnB53QMA3L1+d5HVSJIkSWXBEKVDy6c5nxbtzGYXd28wREmSJEmGKB3a+En9u+fWreSBTa2073dclCRJkmqbIUqHVmqC8/4IgHPqVtHTmxwXJUmSpJpXeIiKiDdFxJqI6IyIOyPiaUc4/6qIeDQi9kXEhoj4WERMOF711px5ZwHwnLo7ALjPLn2SJEmqcYWGqIh4GfBx4EPAucBNwA8jYvEhzv8D4B+A9wGnAa8HXgZ8+LgUXIsWnAfAkmimjl5uX7Oz4IIkSZKkYhXdEvU24Asppc+nlB5OKV0FbAD+7BDnXwrcnFL6akppbUrpeuC/gQuOU721Z8G5UJrEuOhhRWzk9jU72d3RVXRVkiRJUmEKC1ERUQLOB64fdNf1wGWHeNgvgfMj4qL8GsuA5wPfP8zzjI+IKX0bMPmoi68l3Z3Q1QbAuXWPA/DAptYiK5IkSZIKVWRL1CygHmgedLwZmDfUA1JKXwPeA/wyIg4Aq4Cfp5T+4TDP806gdcC28Sjrrlm/NT176+4zREmSJKmGFd2dDyANuh1DHMvuiLgCeDfwJuA84KXACyPiPYe5/oeBqQO2hUdZb20pNcErvwHA6T0PA/BPP3qUji6nOpckSVJtaijwubcDPTy51WkOT26d6vMB4Csppc/nt++PiCbgcxHxoZRS7+AHpJT2A/v7bkfEURdec+aeCcC0feuZSSs7mEpP75A5V5IkSap6hbVEpZS6gDuBKwfddSVwyyEe1ggMDko9ZK1XpqNjZeK0/t3Lx68EYN2OjqKqkSRJkgpVdHe+jwJviIjXRcRpEfExYDHwWYCI+HJEDJy+/DrgzyLi5RFxYkRcSdY69d2UUs9xr75WlJrg/NcCcGbepe/mlduLrEiSJEkqTKEhKqX0deAq4L3APcDTgeenlNblpywG5g94yAeBj+Q/HwK+APwYeOPxqrlmnZCtF/Wi+lsBuHPdriKrkSRJkgpT5JgoAFJKnwE+c4j7rhh0u5tsod33HfvK9AQLLwJgDruYwH6uf6iZ9v0HaBo/ruDCJEmSpOOr6O58qhRTs0kN6wKeEqsAWOu4KEmSJNUgQ5SGZ/wkOO1FALyk/pcA3LF2Z5EVSZIkSYUwRGn4Fl4IwMV1jwDwd999yPWiJEmSVHMMURq+Rdm4qEWxjbp8pvmUXC9KkiRJtcUQpeE74UIoTaIhejk5NgKwYde+gouSJEmSji9DlIavZz90tQHwkpnrAXjex2+yS58kSZJqiiFKo3J56fGiS5AkSZIKYYjS8JWa4A+/DcDy/Q8CMGvSeCaOqy+yKkmSJOm4MkRpZOaeAcD49s3MZzvb2/bzwObWgouSJEmSjh9DlEam1NS/e0ndQwDcvtr1oiRJklQ7DFEamVITXPoXALxh8VYA7li7q8iKJEmSpOPKEKWRW3gBACdt+T4AP3pwK22dB4qsSJIkSTpuDFEauUUXAzAhDjCbrCvfo817i6xIkiRJOm4MURq5KQtg3lkAvG7hZgDuXr+7yIokSZKk48YQpdHJW6Oese0/Afjg9x920V1JkiTVBEOURmfxpQAsiy39h7p7eouqRpIkSTpuDFEanUUXAdm4qKUTOgBY1dJeZEWSJEnScWGI0ug0zuzfPb3rPgBufLSlqGokSZKk48YQpaN2Sd3DANyyanvBlUiSJEnHniFKo1Nqgpf8GwAvr/85AL9eu4uWvZ1FViVJkiQdc4Yojd6iSwAoRTfT2APAr9fuLLIiSZIk6ZgzRGn0Js3u3/3rU7KufL9as6uoaiRJkqTjwhCl0Ss1wbl/CMDlaz4FwDW3rHW9KEmSJFU1Q5SOTr5e1KLYBiQANu3aV2BBkiRJ0rFliNLRWXElAPUBVyxqAOCOdXbpkyRJUvUyROnolJryncRLpz4GwDv/93679EmSJKlqGaI0Zi7igf797p7eAiuRJEmSjh1DlI5OqQle/lUA5uy8o//wg5v3FFWRJEmSdEwZonT0FpwLQN3OVSygBYCbV24vsiJJkiTpmDFE6ehNmNq/+5ELskklblvtoruSJEmqTkcdoiJiSkT8dkScNhYFqQKVmuDytwJwxoFsXNQd63bRvMepziVJklR9RhyiIuIbEfEX+f5E4A7gG8B9EfE7Y1yfKsWiiwGY8ug3CXoAuN3WKEmSJFWh0bREPR24Kd9/CRDANOAvgb8do7pUaRZe0L/75rOzRXf/8mv3ONW5JEmSqs5oQtRUoK+J4XnAt1JKHcD3gRVjVZgqTNNsWHwZAM9/5N39h1NKRVUkSZIkHROjCVEbgEsjooksRF2fH58OdI5VYapAS58KwPLYRKkhANi026+EJEmSqstoQtTHgf8CNgKbgRvz408H7h+bslSRll4OQEP00tPdBcCPH9haZEWSJEnSmBtxiEopfQa4FHgdcHlKqTe/azWOiapt88/p3z0z1gHwi8daiqpGkiRJOiZGNcV5SumOlNK1KaW2iKiPiHOAW1JKN49xfaok9eP6dz9/eRuQTXW+pdWpziVJklQ9RjPF+ccj4vX5fj3wC+AuYENEXDG25alSzWg+mKdvWbm9wEokSZKksTWalqjfBe7N918EnAicSjZW6kNjVJcqUakJ3vh/ANSvv5nJtAPwi8cMUZIkSaoeowlRs4C+2QKeD/xPSukx4AvAWWNVmCrUjJP6d//rWdnkEt+9dzN7Ow8UVZEkSZI0pkYTopqB0/OufM8DfpofbwR6xqowVajxk+CcVwFwxi//AsjWiXpgU2uBRUmSJEljZzQh6kvAN4AHyH5D/kl+/GLgkTGqS5Vs2RUA1EfiuafNBuAV/347HV3dxdUkSZIkjZHRTHF+NfAG4HPAU1NK+/O7eoB/GLvSVLHy9aIAfvMEZ+aTJElSdWkYzYNSSt8c4th/HH05qgoTpvTvPmvcA2Rzj8Dezm4aS6P6ykmSJEllY1TrREXEMyLiuohYGRGPR8R3I+JpY12cKt/EjTf17//ogS0FViJJkiSNjdGsE/UqsskkOoBPAp8G9gE3RMQrx7Y8Vbr6dTfTQDYW6mePtBRcjSRJknT0RtMS9W7g7Smll6WUPplS+kRK6WXAO4D3jG15qkilJnjHBgCiay8/enH2NfvFYy207O0ssjJJkiTpqI0mRC0Drhvi+HfpG/wi1dX37y5uvb1//5crXXhXkiRJlW00IWoD8Kwhjj8rv0/KWqNe8NFs97ZP8dpLTgDgrV+/16nOJUmSVNFGM1XaR4BPRsQ5wC1ka0VdDrwGeMvYlaaKt+wZ/bvPXVrHl27L9ru6e2ksFVSTJEmSdJRGHKJSSv8aEVuBvwJ+Pz/8MPCylNJ3xrI4VbjJ8/t3n9J1DzATgDvW7uTZp88rqChJkiTp6IxqivOU0rUppctTSjPz7XLgBxGxeIzrUyUrNcHFbwRg4g/ezO+dOxeAN3z5Trv0SZIkqWKNKkQdwunAmjG8nqrBsmf27z7z5On9+729qYhqJEmSpKM2liFKerLFF/fvPnXi+v79B7fsKaIaSZIk6agZonRs1R+cQWLi2p/37//0oeYiqpEkSZKOmiFKx8242z7B//vtUwD495vWOC5KkiRJFWnYs/NFxNlHOOWUo6xF1ajUBG+5Hz5xFgBXzNvff9ea7e2csWBqUZVJkiRJozKSKc7vIVsTKoa4r++4swXoyZpm9u9O3XQjsBSAF3zylzz0/ufSWBrNcmWSJElSMUbSne9EYFn+c/C2bMBP6YlKTfCMd2S7P3kX7/3NkwouSJIkSRq9YYeolNK64WzHslhVsBXP7t999rLG/v1Nu/cVUY0kSZI0ak4soeNj9mn9u4u/cBZ19AJw/YPO0idJkqTKYojS8TF+Epz36v6b9fQA8KMHthZVkSRJkjQqhigdPyuu7N/96VsuA+D+Ta083ry3qIokSZKkETNE6fhZcln/7px9K/v3f/ygrVGSJEmqHIYoHT8NE/p3J675Ke99YTZO6p+vf8yFdyVJklQxRhyiIuLuiLhriO3OiLg5Iv4jIn7jWBSrCldqghd+LNu/6SM8e/nk/rs27uooqChJkiRpZEbTEvUjsvWg2oGfAzcCbcBJwK+B+cBPI+LFY1Sjqsnyg+OiFtdt5+ITZwDwnI/dZGuUJEmSKsJoQtQs4CMppaellP4qpfS2lNLTgX8GmlJKzwE+CLxnLAtVlWiccXD/M5fw3FOnF1eLJEmSNAqjCVG/D/z3EMe/lt9Hfv8poy1KVazUBFe+v//m5cum9e8/5ix9kiRJqgCjCVGdwGVDHL8sv6/vuvtHW5Sq3MnP699dWGrv37/x0ZYiqpEkSZJGpGEUj/kU8NmIOJ9sDFQCLgLeAPx9fs5zgbvHpEJVn6kL+3frV/0EWATAx3/6OH/8tBNpGj+uoMIkSZKkIxtxS1RK6YPAH5MFp0+ShaqLgD9OKX0oP+2zwIvGqkhVr/Grf8Jt73xm/+1H7dInSZKkMjeqdaJSSv+VUro0pTQj3y5NKX11wP37Ukqdh7uGalipCf7459n+yp8ypWdn/10v/cytztInSZKksjbqxXYj4vyIeFVE/EFEnDuWRakGzDq5f7dx/f/xiZef03+7pzcVUZEkSZI0LKNZbHdORPyMbDzUJ4FPA3dGxA0RMXusC1SViji4/+0/5elLG/tvnnX19bZGSZIkqWyNpiXqU8AU4Iy8K9904Mz82CfHsjhVsVIT/NF3+29OLyV+9/wTCixIkiRJGp7RhKjnAX+WUnq470BK6SHgz4HfHKvCVANOOO/g/j+dyLNXTOm/uavDGfIlSZJUnkYTouqAA0McPzDK66lWjZ8M5/5R/83LT5rRv//rNbuKqEiSJEk6otGEnp8Bn4iIBX0HIuIE4GPADWNVmGrEqc/v363rPTgO6qqv3+u4KEmSJJWl0YSovwAmA2sjYlVErATW5Mf+ciyLUw1Ycln/buOW27n2TQdvt+4bqsFTkiRJKtZoFtvdkFI6D3gB8HGyySSen1I6P6W0YawLVJWrazi4/9B3OGXe5P6bl374Z7ZGSZIkqeyMegxTSuknKaVPpZQ+mVL6aUQsiogvjmVxqgGlJviDb2X7932dxp69vPXZK4qtSZIkSTqMsZwIYgbw6jG8nmrFoosO7q/+Bc86bU7/zcea9xZQkCRJknRozqan4k2YAhe8Ptv/n1ezbOrBhXivf7C5oKIkSZKkoRmiVB5O/62D+92d/bufuXEVbZ1OMCFJkqTyYYhSeTjh/P7dxo+fzJ3/36X9t+9c55pRkiRJKh8NRz4lExH/e4RTph1lLapl4yfDxW+E2/8NgInj6vvvevWXfs1D738ujaVhf10lSZKkY2YkLVGtR9jWAV8e6wJVQ5Y/t3+3sWs3X3n9wQkn2vc71bkkSZLKw7D/tJ9Seu2xLERi8cUH99ffwnmn/W7/zQs/dIOtUZIkSf9/e/ceJVlZ2Hv/98BYg9MzoAYQ0CBGRMVLNKAmGq8InGiWJ5oT9U2yEs/RJOpJXBrMEbwgURSNCnjLVXM83o7GN9EcQV8vJN6N8Y6iHjARgtwEwYGZgRnE/f5R1V3V1dXdu7rrXp/PWrWovWtX9W7di+E7z7OfYiK4J4rJUdqr8uWDz8pC2ZfnPvbo8Z0PAAD0IKKYHI2F5Bkfa2/v3ZUTj21/Z9T3fug7owAAGD8RxWQ59L7t568/Jve4Q/sS/cg3rx7DCQEAwHIiismydXvy8Of1fOkvPvXv2XnzvhGfEAAALCeimDzH/MrS0217rsrXTz9xaftT//facZwRAAAsEVFMnsPv337+vY/nDtsaeeYj7p4kee57v549+yx3DgDA+IgoJtvHXpLs251ffcDhS7suvsYCEwAAjI+IYvI0FpLnfqO9/aN/yzF33rG0ed43rhrDSQEAQJOIYjJtP6T9/K8ekezbvbT51s9+Pz/eY4EJAADGQ0QxmRoLyX9+y9Lmttvtnwtf1l5g4oLvXDOOswIAABHFBDvm5Pbzs+6SLbfdvLR5yvsvtMAEAABjIaKYXAuHJPd/ytLmtsaWXHDKI5e2//3aXeM4KwAA5pyIYrLd+1fbz/fckMMPuv3S5q++6XNGowAAGDkRxWS7x2Pazy//l2xrbMm5T/v5pV3X7947hpMCAGCeiSgm2377t5///TOSfbtzwr3vvLTr0xdfN4aTAgBgnokoJltjIXnmJ9rbe67P/vuVpc0XfeBb2b331jGcGAAA80pEMfkOPbb9/Nz7ZVv25nMvbE/z+9p//HgMJwUAwLwSUUy+xkJy8qva21WVOy40ljZ/+23/aoEJAABGRkQxHe75n9rP/+OL2dbYkg8852FLu449/aNCCgCAkRBRTIcDD2s/v+gfkiT3OmzHmE4GAIB5JqKYPl9/V7Lr2mxrbMnrfuMBS7stdw4AwCiIKKZDYyE57Yr29nc/lCQ5+b7tEap//u61oz4rAADmkIhiepT20uY57/nJvt3Lljt/6T9eZLlzAACGTkQxPRoLyXO/1t6+7pJsa2zJF0577NKu+77sYxaYAABgqEQU02X7ndvP//pRyb7dOfyg2+d3fulu4zsnAADmiohiujQWkif9VXv75uYX7T7l+Lsu7frqZdeP+qwAAJgjIorpc/dHtZ//2z8nSX7ukO1Lu97zr5eP+owAAJgjIorpc8CB7ef/578n+3Yve/nD37w6F19z44hPCgCAeSGimD6NheRZn2tv7/xBtjW25NsvP3lp10nnfMYCEwAADIWIYjrd6e7t5295SLJvd7Y1tuRvn3780u4bdu8bw4kBADDrRBTTqbGQ/Prb2ts/2Zskeejd77S06+Gv+WejUQAADJyIYnrd88T28z+7e7Jvdxa23i6vfvL9lnb/8KZbxnBiAADMMhHF9DrgoOThz29v79uTJHnCA45Y2vWBr1456rMCAGDGiSim2wOe2n5+zbeSJPvvV5Z2veGCS3Kt0SgAAAZIRDHd7nhk+/k7f21pgYkLX9ae6vfgV17g3igAAAZGRDFbbr4hSXLg7Rt55ZPa90bdvO+2cZ0RAAAzRkQx3RoLyWlXtLfPue/Sl++ecO9Dlna/+4uXjfrMAACYUSKK6bd1e3LSme3tvbuSNEejFp398Utyw+69oz4zAABmkIhiNtz7ie3n3/90kmRbY0u+fnr73qi//+oV3e8CAIC+iShmw/aD28//4ZlLU/oaW9qX+Jnnfyc/3rNv1GcGAMCMEVHMph9+O0lzNOprL33c0u4HvvzjVuoDAGBTRBSzobGQvKjji3Xf+ril0ag7LmzNKSfdc+mlG282GgUAwMaJKGZHYyH53fPa2686YimknvSguyztPu/Cq0Z9ZgAAzBARxWy5yy/03H2nha1Lz888/7u53kp9AABskIhitjQWkie+pb2967okK++N+oVXfMK9UQAAbIiIYvYc3Y6lXPQPS0+33m7/ZYcde/pHhRQAAH0TUcyeA3a0n19wRrL3piTN0agLX3Zi7/cAAEBNIorZ01hIXnBJe/usuy4tMHHg7Rt57X+5/9JLl1+/e9RnBwDAlBNRzKbGwqovPf7+hy89P/ncz5rSBwBAX0QUs6mxkPzBZ9vbl/3L0tNSyrJDL79+z6jOCgCAGSCimF0/83Pt5+9+8tKUvm2NLfn2y09eeunkcz9jNAoAgNpEFPPjuvZ9UtsaW/LOZzx4afsbl98wjjMCAGAKiShmV2MhedGV7e2/ftTSaFSS3OfwA5ee/z9/869GowAAqEVEMV9edcSyaX2dfG8UAAB1iChmW/doVIfue6OS5Ce3/XQUZwUAwBQTUcy+xkLy1He3t6///tLTbY0t+fypj1nafsCfftxoFAAAaxJRzIe7PqT9/C8fvuzeqCPusC2n/sq9lrav/LElzwEAWJ2IYj5s7fry3T0/Wrb5pAfdZen548625DkAAKsTUcyHxkJy2hXt7XPvv2w0ascBt1t2uEUmAABYjYhifpSyfLtrpb7uRSaqqhrVmQEAMEVEFPOjsZCc9oNVX97W2JLz/ujhS9sf/PoVqx4LAMD8ElHMl607kie+ub29c/ny54cddMDS8xd/4KJct+uWUZ0ZAABTQkQxf+55Uvv5W45fdm9U9xfwHn/mBe6NAgBgGRHF/Nm6ffl2x0p9ve6NssgEAACdJiKiSinPKeshFOYAACAASURBVKV8v5RySynlK6WUR6xx7CdLKVWPx/mjPGem2Dor9W1rbMlFf3rSsrfs2nvrqM4OAIAJN/aIKqU8Ncm5SV6Z5EFJPpPkI6WUI1d5y5OTHN7xuF+S25K8f/hny8zoXqnvx5cv21zYert86A8ftrT9kFf+k9EoAACSTEBEJfnjJG+rquqtVVV9p6qq5yW5PMmzex1cVdX1VVVdvfhIcmKSPRFR9KOxkJxySXv7zx+6bDQqSe5x6I5l21fvtMgEAABjjqhSSiPJcUk+1vXSx5I8bOU7enpGkvdWVbW714ullK2llAMXH0l29DqOObR1Yfn2rh8u29zW2JJvntGe1vfY138qu03rAwCYe+MeiTo4yf5Jrunaf02Sw9Z7cynlIWlO53vrGoedlmRnx2P1LwpivnTfG/XGB64Yjdp/v+XT/u77so+Z1gcAMOfGHVGLqq7t0mNfL89I8q2qqv51jWPOSnJQx+OuGzpDZlP3vVGvOmLFIhPdq/Vd+eObR3FmAABMqHFH1HVpLgrRPep0aFaOTi1TStmW5GlZexQqVVXtrarqxsVHkps2cb7MmsZC8qLlX7ibfXuWbW5rbMnnT33M0vbjzv600SgAgDk21oiqqmpfkq+kuThEpxOTfH6dtz8lydYk7xrCqTFPGgvJf/1oe/vylQObd9jWWLbtu6MAAObXuEeikuTsJM8spfy3Usp9SinnJDkyyV8mSSnlHaWUs3q87xlJPlhV1Y96vAb9Ofz+7efv+81k7/IBy17T+q650bQ+AIB5NPaIqqrqfUmel+T0JF9P8sgkj6+q6rLWIUem+X1QS0opxyT55SRvG+GpMk/OuuuKRSa2Nbbksy989NL2Y15nWh8AwDwae0QlSVVVf15V1VFVVW2tquq4qqo+3fHao6uqenrX8RdXVVWqqvr4yE+W2dTr3qg9N6w47E4LW5dtm9YHADB/JiKiYCI0FpLnXtje/vrK2+16TesTUgAA80VEQaftB7eff/KsZOcVKw7Z1tiSL7/khGX7RBQAwPwQUbCWc45dcW9U0gypTsefeYGQAgCYEyIKOvW6N6rrC3iT3qNRpvUBAMwHEQXdGgvJaV3T+Lq+gDdZORqVCCkAgHkgoqCXrduT3/lQe/uyld/93GuRicT9UQAAs05EwWoOvU/7+ft/J7l55ZLnvULK/VEAALNNRMFqGtuWb7/mqFUXmfjSix+7bJ9pfQAAs0tEwWpqLjKRJAtbb7di3+69tw7rzAAAGCMRBWvpY5GJ7ml9D37lPxmNAgCYQSIK1rN1e/K757W3X3f0qtP6LHsOADD7RBTUcZdfWL79k709D7PsOQDA7BNRsBF/dvdVR6N6LXu+y/1RAAAzQ0RBHb0Wmbj2kp6H9gqph7g/CgBgZogoqKuxkJzSEU5/86ieo1GJ+6MAAGaZiIJ+bF1Yvr3KkudJ7/ujrrhh5cp+AABMFxEF/ejju6N6Tes78ZzP5KhTzzciBQAwxUQU9KuxkPzJ95bv6/HdUUnvaX2JqX0AANNMRMFGLBySPOVd7e3Lv7jqoQdvP6Dnin0iCgBgOoko2Kgjf7H9/H2/ley+btVDe03tO/7MC4QUAMAUElGwUY1ty7dfe49VF5lImiH1pRc/dtk+0/oAAKaPiIKN6mORiUULW2+3Yp+QAgCYLiIKNqNXSN28c9XDe03rS4QUAMA0EVGwWY2F5Nn/0t4+5z7rTusTUgAA00tEwSDc8cjl2+tM61tt6XMRBQAw+UQUDEJjITntiuX7dl+/5lu2Nbas2GfFPgCAySeiYFC2bk/++5fb22+437qjUZe++gkrRqRM6wMAmGwiCgbpoCOWb68zrS/pPSJ17OkfzXW7bhnkmQEAMCAiCgZpA8uer7bQhKl9AACTSUTBoDUWkhd8b/m+fXvWfIsV+wAApoeIgmHYfkjym+9vb7/u6FrT+nqt2CekAAAmi4iCYTnq4cu3a9wfdfD2A4xIAQBMOBEFo7R317qHGJECAJhsIgqGpdciE68/Zt3RqKT3in2JkAIAmAQiCoap1yITNZc9v/TVTzC1DwBgAokoGLbGtpX7aoRUYmofAMAkElEwbL2m9SW17o9KTO0DAJg0IgpGYZP3R5naBwAwOUQUjMoG749aZGofAMBkEFEwSpu4PyrxPVIAAJNARMEoNRaSM3auHJG65cbaH7HWiNR1u27Z7BkCALAOEQXjsP2Q5OkfaW+ffe/ao1HJ6otNHH/mBUakAACGrFRVNe5zGKlSyoFJdu7cuTMHHnjguE+HebZvd3MqX6cXXdkcrappz76f5NjTP9rztW+//ORVYwsAgOTGG2/MQQcdlCQHVVVVe2qQkSgYl14r9l3+pb4+YrWpfUmMSAEADImRKBi3G69Ozr7X8n1GpAAAhs5IFEyrA3as3NfHin1Jc0Sq16p9iZX7AAAGTUTBuPWa1pck+/b09TFCCgBgNEQUTIJeIfW6o/sajUrWDylLoAMAbJ6IgknRWFj5/VF9TutL1g6p48+8IEeder5RKQCATRBRMEka21buu/aSvj9mrZBKrNwHALAZVueDSfSjf0/e9KDl+/pcsW/RdbtuyfFnXtDzNSv3AQDzzOp8MEt23Hnlvg1M7UuSg7cf4D4pAIABElEwiVZbsW9v/xGVrP2lvMefeYHpfQAAfTCdDybZvt3NEahOG5zWl6z9pbyJ6X0AwHzZ6HQ+EQWTbte1zeXOu20iptwnBQDgniiYXb1W7Ev6/jLeTmtFkvukAADWZiQKpsXeXclZd1m+bxOjUYnpfQDAfDMSBbNu6/bklIuX79vgin2L1vs+KaNSAAArGYmCadJroYlk0yNSydr3SSVGpQCA2WMkCubBakufb3JEKln7+6SS5qiUpdABAIxEjft0YGOGOCKVGJUCAOaDkSiYJ0MckUqMSgEArEVEwbQackjVWXTiqFPPF1MAwNwRUTDNxhxSiVEpAGD+iCiYdiMIqUtf/YR8+SUnrHqMpdABgHliYQmYFUNebCJZ/8t5E4tOAADTY6MLS4gomCUjCKlk/dX7EjEFAEw+EVWTiGLmjSik6oxKJWIKAJhcljgHmhoLyQu+t3L/q45Idl07sB9TZ9GJxMITAMDsMRIFs2pEI1KJUSkAYDqZzleTiGKurBZSL/hesv2Qgf84MQUATBPT+YCVVlv+/HVHD2T58251lkNPTPEDAKabkSiYB6uNSCVDmd6XGJUCACafkShgdauNSCXJvj1D+ZH9LDxx1KnnG5kCAKaGkSiYJ2MYkVpU57ulEiNTAMDoGIkC1rfWiNSAl0DvdvD2A2qPTF2365ahnQcAwGaJKJg3a4XUkBacWFR3it/xZ16Qo049X0wBABPJdD6YV2tN7RvSEujd6k7xS0zzAwAGz3Q+oD+NheSMnc1g6jbkEalFdaf4JZZFBwAmh5EomHdrjUglQ19wYlHdJdETo1IAwGBsdCRKRAFNu65tjkD1MqKQSvqLqS+/5IQcvP2AIZ8RADCrRFRNIgrWMMYl0Lv1E1OJ0SkAoH8iqiYRBeuYoJBKxBQAMDwiqiYRBTVMyH1SncQUADBoIqomEQV9mJD7pDr1syx6IqYAgNWJqJpEFPRhAkekFvU7MmURCgCgm4iqSUTBBkzAF/OuRkwBABslomoSUbBBEzwqlfQfU4mgAoB5J6JqElGwSavF1CmXJDsOHf35dBFTAEBdIqomEQUDMIELTvTS7yIUiaACgHkiomoSUTAgE3yfVLeNxFRiZT8AmHUiqiYRBQM04fdJddvIVL/E6BQAzCoRVZOIgiGYolGpZOMxlRidAoBZIqJqElEwJGvdJ5VM3KjUIqNTADC/RFRNIgqGaMqm93XazOiUoAKA6SSiahJRMAJTOiq1SFABwHwQUTWJKBiRKR6VWrSZmErcPwUAk05E1SSiYITWC6kJXHRiNYIKAGaPiKpJRMEYzMCoVCdBBQCzQUTVJKJgTGZoVGrRZmMqEVQAME4iqiYRBWM25YtOrEZQAcD0EVE1iSiYAOuNSiVzHVOJVf4AYBREVE0iCibIeqNSUzjFr5OgAoDJJqJqElEwYWZ4VKrToILKlD8AGBwRVZOIggk146NS3a7bdUuOP/OCTX+OUSoA2DgRVZOIggk3owtPrGZQI1SJoAKAfomomkQUTIE5meLXbZBBlZj6BwDrEVE1iSiYInM2xa/ToIMqMVIFAN1EVE0iCqbMnI5KdRtGVBmpAmDeiaiaRBRMoTohlcxFTCXDCapEVAEwf0RUTSIKptwcT/HrZVhBlYgqAGafiKpJRMEMMMVvVaIKAOoTUTWJKJgh641KJXMbU4sG9X1UvYgqAKadiKpJRMEMElO1DHOUKhFVAEwfEVWTiIIZZfGJvg07qhJhBcBkE1E1iSiYcUalNmwUUZUIKwAmh4iqSUTBnKgzMjVnK/n1S1QBMOtEVE0iCuaMmBqYUUXVl19yQg7efsDQfw4AiKiaRBTMIfdLDcWooioRVgAMh4iqSUTBnHPP1FCNMqxMAwRgs0RUTSIKqD0yZZrfpo0yqhJhBUB/RFRNIgpYIqbGYphfANyLsAJgNSKqJhEFrFA3phJT/YZg1KNVi8QVACKqJhEFrMoCFBNhXFGVCCuAeSOiahJRwLqMTE2cUU8B7GRlQIDZJaJqElFAbWJqoo0zrBYZuQKYbiKqJhEF9K3OsuiLxNTYjHMaYDdxBTAdRFRNIgrYMDE1dSYprBJxBTBpRFRNIgrYNNP8ptqkhVUirgDGRUTVJKKAgRFTM2US4yqxsAXAMImomkQUMHD9xFQiqKbMJCxgsRajWAAbJ6JqElHA0IipuTGpo1adxBXA+kRUTSIKGIl+FqF4wfeS7YcM93wYiWmIq05CC5h3IqomEQWMVD+jU2Jqpk36tMBeRBYw60RUTSIKGAtT/VjFtI1eLRJYwCwQUTWJKGCs+o2pRFDNsWkcvVoksoBpIKJqElHAxDA6xQZN6+hVN8u3A+MmomoSUcDEEVMM0DSPXnUzmgUMm4iqSUQBE8tUP4ZsVkawuoktYKNEVE0iCpgK/SyRnogpBmJWIysRWkBvIqomEQVMlX5jKhFUDMUsB9YioQXzR0TVJKKAqWV0igk1D4G1SGjBbBFRNYkoYOq5d4opM0+RtcjKgzAdRFRNIgqYKab7MQPmMbIWiS0YLxFVk4gCZpbpfsyoeY6sbqYTwmCJqJpEFDDzjE4xh4RWb6IL1iaiahJRwFwRVLBklr6IeNBMK2ReiaiaRBQwtwQV1CK26jHKxSwQUTWJKGDubWR1v0RQQYupgxtnxItJI6JqElEAHTYSVC/4XrL9kOGcD8wYo1qbY7SLYRNRNYkogFUIKhgbsbV5gouNEFE1iSiAdZjuBxPLVMLBEl6IqJpEFEAfBBVMJbE1Gu7xmn4iqiYRBbBBggpmkuAaHxE2flMbUaWU5yT5kySHJ7koyfOqqvrMGsffIckrkzw5yR2TfD/JKVVVfbjmzxNRAJu10aBKRBVMMcE1mUxL3LipjKhSylOTvDPJc5J8LskfJHlmkmOrqvqPHsc3Wsf9MMmrkvwgyc8muamqqm/U/JkiCmCQBBWwBotmTId5DbFpjagvJvlqVVXP7tj3nSQfrKrqtB7HPyvNUat7V1V16wZ/pogCGJaNBpVV/oAWo12zYVqmKk5dRLVGlfYk+Y2qqj7Qsf8NSR5YVdWjerznw0mub73vPye5Nsl7krymqqrbVvk5W5Ns7di1I8kPRBTAkAkqYISMeE2fSRj92mhEjfOsD06yf5JruvZfk+SwVd7zc0kem+TdSR6f5J5J3pLm7/HyVd5zWpKXbfZkAehTYyE5Y2fzeT9B9bqjl2+b8gfUcPD2A3Lpq5/Q9/uMfLERkzDxsXsorPTYt2i/NO+H+v3WyNNXSilHpDnFb7WIOivJ2R3bO9K8lwqAUdloUCXLjxVUwIBta2zZUHwlAmyzdu/9ydhHojZqnGd9XZLbsnLU6dCsHJ1adFWSW7um7n0nyWGllEZVVfu631BV1d4kexe3SymbOmkANmlQQZWIKmCsBNjm7NnX826cqTC2iKqqal8p5StJTkzygY6XTkzyj6u87XNJfrOUsl9VVT9t7TsmyVW9AgqACdcZVEmy69qV0/nWYpQKmFKbCbBkNiJs+9bpHIVKxr863+IS589K8oUkv5/k95Lct6qqy0op70hyxeJKfaWUn03y7SRvT/KmNO+J+tskb6yq6pU1f6bV+QCmQb9B1UlQAaxpEhbisLDEBlVV9b5Sys8kOT3NL9v9VpLHV1V1WeuQI5P8tOP4y0spJyU5J8mFSa5I8oYkrxnpiQMwfNsPMe0PYEg2uhDHolkYCduMsY5EjYORKIApt5kv97V8OgAdpu57osZFRAHMGNP+ANggEVWTiAKYYZsJqkRUAcwZEVWTiAKYE5uZ9rdIVAHMNBFVk4gCmFOiCoAuIqomEQXAQILKIhUAU09E1SSiAFjBKBXAXBJRNYkoANa12QUqjFIBTAURVZOIAqAvRqkAZpaIqklEAbAp7qcCmBkiqiYRBcBADSKqEqNVAGMgomoSUQAMzaCCykgVwEiIqJpEFAAjtdlFKhYZqQIYOBFVk4gCYGyMVAFMFBFVk4gCYGK4nwpgrERUTSIKgIllpApgpERUTSIKgKkxqKhKjFYB9CCiahJRAEw131MFMDAiqiYRBcBMcV8VwIaJqJpEFAAzbVBLqifCCph5IqomEQXA3DFaBdCTiKpJRAEw96wCCJBERNUmogCgi1UAgTklomoSUQBQgymAwBwQUTWJKADYIMurAzNGRNUkogBgQNxbBUw5EVWTiAKAIRJWwBQRUTWJKAAYMfdXARNKRNUkogBgAgziS4FFFbBJIqomEQUAE2oQYWUaINAHEVWTiAKAKeH+KmDIRFRNIgoAptQgvxRYWAERUbWJKACYMYOYBrjIfVYwV0RUTSIKAOaAsAJqEFE1iSgAmFODDCvTAWEmiKiaRBQAsGSQYZUYtYIpI6JqElEAwJqEFcwNEVWTiAIA+jbIlQETYQUTQkTVJKIAgIEYdFgl4gpGTETVJKIAgKEyHRCmhoiqSUQBACMnrGAiiaiaRBQAMBHcZwVjJ6JqElEAwETzfVYwMiKqJhEFAEwdo1YwFCKqJhEFAMwEqwPCpomomkQUADDTjFpBbSKqJhEFAMylQceV+62YASKqJhEFANBiSiBzTkTVJKIAANYhrpgTIqomEQUAsAHDCKvEtEDGSkTVJKIAAAZokN9r1UlcMQIiqiYRBQAwAsOKq8TUQAZGRNUkogAAxmRYUwIXGb2iTyKqJhEFADBhhh1XRq5YhYiqSUQBAEyRYU4LTATWnBNRNYkoAIApN+ywMi1wboiomkQUAMAMGnZYLTJyNVNEVE0iCgBgjowqrhKBNYVEVE0iCgCAJEavEFF1iSgAANY07NUCF7n3auxEVE0iCgCADTN6NVNEVE0iCgCAgTN6NZVEVE0iCgCAkRvFCJbRq76JqJpEFAAAE8Ho1diJqJpEFAAAE29UgTXno1ciqiYRBQDA1DM9cCBEVE0iCgCAmTWq1QNnZIqgiKpJRAEAMJcsz76CiKpJRAEAQJc5vQdLRNUkogAAoA/DCqznX5QcdNfBf24fNhpRW4Z3SgAAwNRrLCRn7Oz92mamCN66Z+PnNGYiCgAA2Jjth6weWOuNYO04fDjnNAIiCgAAGLy1RrCm3H7jPgEAAIBpIqIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6IKIAAAD6sGXcJzAuN95447hPAQAAGKONNkGpqmrApzLZSil3SfKDcZ8HAAAwMe5aVdUVdQ+ex4gqSY5IctMmPmZHmiF2101+DvPLNcRmuYbYDNcPm+UaYrMm6RrakeTKqo8wmrvpfK3/cWpXZi/NDkuS3FRVlXmB9M01xGa5htgM1w+b5RpisybsGur751tYAgAAoA8iCgAAoA8iamP2JvnT1j9hI1xDbJZriM1w/bBZriE2a6qvoblbWAIAAGAzjEQBAAD0QUQBAAD0QUQBAAD0QUQBAAD0QURtQCnlOaWU75dSbimlfKWU8ohxnxPjV0o5rZTypVLKTaWUH5ZSPlhKuVfXMVtLKW8qpVxXStldSvk/pZS7dh1zZCnlQ63XryulvLGU0hjtb8O4ta6nqpRybsc+1w9rKqXcpZTyrlLKj0ope0opXy+lHNfxeimlnFFKubKUcnMp5ZOllPt2fcYdSynvLKXsbD3eWUq5w+h/G0atlLKllHJm679xbi6l/Hsp5fRSyn4dx7iGWFJKeWTrz5wrW39m/VrX6wO5Xkop9y+lfKr1GVe0rsuSMRJRfSqlPDXJuUlemeRBST6T5COllCPHemJMgkcleUuSX0xyYpItST5WSlnoOObcJE9K8rQkv5xke5LzSin7J0nrn+cnWWi9/rQkv57k9SP6HZgApZQHJ/n9JBd2veT6YVWllDsm+VySW5P8SpJjk5yS5Mcdh/2PJH+c5A+TPDjJ1Uk+XkrZ0XHMe5I8MMl/aj0emOSdwz5/JsILkzwrzevjPmleL3+S5I86jnEN0WkhyTfSvB562fT1Uko5MMnHk1zZ+ow/SvKC1ueOT1VVHn08knwxyV907ftOkrPGfW4ek/VIckiSKskjW9sHJdmX5KkdxxyR5LYkJ7e2f6W1fUTHMU9LckuSA8f9O3mM5LrZnuTiJI9L8skk57p+PGpeO69O8pk1Xi9Jrkrywo59W9OMrD9obd+n9e+th3Yc84utffca9+/oMfRr6Lwkb+va9/dJ3tl67hryWOv6qZL8Wsf2QK6XJM9uvWdrxzGnJrkira9rGsfDSFQfWlNijkvysa6XPpbkYaM/IybcQa1/Xt/653FJbpeO66eqqiuTfCvt6+eXknyrtX/RR9P8l85xYR68Jcn5VVV9omu/64f1PDHJl0sp729NKf5aKeX3Ol6/e5LDsvwa2pvkU1l+De2squqLHcf8S5Kd8efcPPhskhNKKcckSSnl59Mc1f5w63XXEP0Y1PXyS0k+1Xrvoo+m+ReJRw3r5NezZVw/eEodnGT/JNd07b8mzYsEkjTnACc5O8lnq6r6Vmv3YUn2VVV1Q9fhndfPYem6vqqquqGUsi+usZlXSnlamrFzfI+XXT+s5+fS/Bvbs5O8KslDkryxlLK3qqp3pH0N9Poz7G6t54cl+WGPz/5hXEPz4DVp/gXgd0spt6X53zwvrqrqf7dedw3Rj0FdL4clubTHZyy+9v1NneUGiaiNqbq2S499zLc3J3lAmn+Dt57u66fXteQam3GllJ9N8oYkJ1VVdUs/b43rh6b9kny5qqoXtba/1rqB+9lJ3tFx3Hp/hrmG5tdTk/x2kt9MclGa96acW0q5sqqq/9VxnGuIfgzieun1Gau9dyRM5+vPdWneb9D9NymHZmVlM6dKKW9Kc1rNY6qq+kHHS1cnabRu/u7Uef1cna7rq3X87eIam3XHpXktfKWU8pNSyk/SXKzkua3n18T1w9quSvLtrn3fSbK48NHVrX+u9WfY1Unu3OOzD4lraB68Nsmrq6p6b1VV36yq6p1JzklyWut11xD9GNT1suLPttZnJGO8pkRUH6qq2pfkK2muvNbpxCSfH/0ZMUlay3i+OcmTkzy2qqru4eWvpLlq1okd7zk8yf3Svn6+kOR+rf2LTkqyt/V+ZtcFSe6f5t/8Lj6+nOTdHc9dP6zlc0nu1bXvmCSXtZ5/P83/GOm8hhppxnrnNXRQKeUhHcc8NM0pXv6cm33bkvy0a99taf/3omuIfgzqevlCkkd2fV3HSWmu1nfpsE5+XeNeyWPaHmkOde9L8t/SXFHknCS7ktxt3OfmMfZr48/TXD3mUWn+jcni4/Ydx/xFksuTnJDmEvkXJPl6kv1br++f5JtJPtF6/YTW8W8a9+/nMZZr6pNprc7n+vGocb08OM3QflGSo9OckrU7yW91HPPC1r+nnpRmgL8nzf8Q2dFxzEfSXLL4F1uPC5N8aNy/n8dIrqG3J/lBkiekecP+k5Jcm+Q1riGPVa6Z7Wn/xV+V5Pmt50cO6npJM6iubr33fq3P2pnklLH+7uP+H38aH0mek2b5Lv7t7iPHfU4e43+0/uXR6/H0jmMOSPKmJD9KsifJh5L8bNfnHJnmMrN7Wse9KR3LenrMz6NHRLl+PNa7Zn41zZC+Jc2pfL/X9XpJckaaU/9uSXOVrPt1HXOnJO9KcmPr8a4kdxj37+YxkutnR5rfR3dZkpuT/FuSM5M0XEMeq1wzj17lv33ePsjrJc2ZGp9ufcZVSV6WMS5vXlVV84cDAABQj3uiAAAA+iCiAAAA+iCiAAAA+iCiAAAA+iCiAAAA+iCiAAAA+iCiAAAA+iCiAAAA+iCiAJhZpZRLSynPG/d5ADBbRBQAU6+U8vRSyo97vPTgJH89gp8v1gDmyJZxnwAADEtVVdeO+xz6UUppVFW1b9znAcDajEQBMDCllE+WUt5YSvmzUsr1pZSrSyln1HzvQaWUvy6l/LCUcmMp5Z9KKT/f8frPl1L+uZRyU+v1r5RSji+lPDrJ/0xyUCmlaj3OaL1n2QhR67U/KKWcV0rZU0r5Tinll0opR7fOfXcp5QullHt0vOcepZR/LKVcU0rZVUr5UinlcZ2/c5K7JTln8ed3vPbrpZSLSil7W+dyStfvfGkp5SWllLeXUnYm+ZtSSqOU8uZSylWllFtax5zW1/8RAAyViAJg0H43ye4kn1SioAAAA/pJREFUD03yP5KcXko5ca03lFJKkvOTHJbk8UmOS/LVJBeUUu7UOuzdSX6Q5hS945K8OsmtST6f5HlJbkxyeOvxujV+3EuTvCPJA5N8N8l7kvxVkrOSHN865s0dx29P8uEkj0vyoCQfTfKhUsqRrdef3Dqv0zt+fkopxyX5uyTvTXL/JGckeUUp5eld5/MnSb7V+p1ekeS5SZ6Y5ClJ7pXkt5NcusbvA8CImc4HwKBdWFXVn7aeX1JK+cMkJyT5+BrveUyaoXFoVVV7W/teUEr5tST/Jc37mo5M8tqqqr67+NmLb26N4lRVVV1d4/z+Z1VVf9d632uSfCHJK6qq+mhr3xvSHNlKmh/6jSTf6Hj/S0opT0ozdN5cVdX1pZTbktzU9fP/OMkFVVW9orV9cSnl2DSj6e0dx/1TVVVL0deKs0uSfLaqqirJZTV+JwBGyEgUAIN2Ydf2VUkOXec9x6U54vOj1pS5XaWUXUnunmRxat3ZSd5aSvlEKeXUzil3mzi/a1r//GbXvgNKKQcmSSlloTU98dullB+3zuveaUbdWu6T5HNd+z6X5J6llP079n2565i3pzlK9n9bUyNPWvc3AmCkRBQAg3Zr13aV9f+82S/N2Hpg1+NeSV6bJFVVnZHkvmlO+3tskm+3RoQ2c37VGvsWz/m1SX49yYuTPKJ1Xt9M0ljn55SOz+rc121350ZVVV9NMx5fmuT2Sf6ulPL/rvOzABgh0/kAmARfTfN+qJ9UVXXpagdVVXVxkovTXMThfyf5r0k+kGRfkv1Xe98mPSLJ26uq+kCSlFK2Jzmq65heP//bSX65a9/DklxcVdVta/3AqqpuTPK+JO9rBdT/V0q5U1VV12/sVwBgkIxEATAJPpHmvUkfLKWcXEo5qpTysFLKma0V+G7fWrHu0aWUu5VSHp7mAhPfab3/0iTbSyknlFIOLqVsG+C5fS/Jk0spD2ytFvierPzz89Ikjyyl3KWUcnBr3+uTnFBKeWkp5ZhSyu8m+cOsvehFSinPL6U8rZRy71LKMUl+I8nVSXp9DxYAYyCiABi71gIKj0/y6SR/m+Zo03vTHPG5JsltSX4mzVX1Lk5z1buPJHlZ6/2fT/KXaY7eXJvmqoCD8vwkN6S5CuCH0lyd76tdx5zeOtd/a/38xWl5T0nytDRX33t5ktOrqnr7Oj9vV5IXpnmv1Jdan/v4qqp+uunfBICBKM0/twAAAKjDSBQAAEAfRBQAQ1dK+a3Opcu7HheN+/wAoB+m8wEwdKWUHUnuvMrLt1ZV5QtlAZgaIgoAAKAPpvMBAAD0QUQBAAD0QUQBAAD0QUQBAAD0QUQBAAD0QUQBAAD0QUQBAAD04f8HqhFBfxEWvbAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cvresult = pd.DataFrame.from_csv('2_nestimators.csv')\n",
    "\n",
    "cvresult = cvresult.iloc[25:]\n",
    "# plot\n",
    "test_means = cvresult['test-mlogloss-mean']\n",
    "test_stds = cvresult['test-mlogloss-std'] \n",
    "        \n",
    "train_means = cvresult['train-mlogloss-mean']\n",
    "train_stds = cvresult['train-mlogloss-std'] \n",
    "\n",
    "x_axis = range(25,cvresult.shape[0]+25)\n",
    "        \n",
    "fig = pyplot.figure(figsize=(10, 10), dpi=100)\n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'n_estimators_detail.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n"
     ]
    }
   ],
   "source": [
    "nestimators = pd.read_csv(\"2_nestimators.csv\")\n",
    "best_n_estimator = nestimators.shape[0]\n",
    "print(best_n_estimator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>test-mlogloss-mean</th>\n",
       "      <th>test-mlogloss-std</th>\n",
       "      <th>train-mlogloss-mean</th>\n",
       "      <th>train-mlogloss-std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>995</td>\n",
       "      <td>0.598319</td>\n",
       "      <td>0.002226</td>\n",
       "      <td>0.553618</td>\n",
       "      <td>0.001038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>996</td>\n",
       "      <td>0.598294</td>\n",
       "      <td>0.002224</td>\n",
       "      <td>0.553553</td>\n",
       "      <td>0.001036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>997</td>\n",
       "      <td>0.598272</td>\n",
       "      <td>0.002231</td>\n",
       "      <td>0.553501</td>\n",
       "      <td>0.001030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>998</td>\n",
       "      <td>0.598250</td>\n",
       "      <td>0.002230</td>\n",
       "      <td>0.553437</td>\n",
       "      <td>0.001030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>999</td>\n",
       "      <td>0.598229</td>\n",
       "      <td>0.002231</td>\n",
       "      <td>0.553381</td>\n",
       "      <td>0.001027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     n_estimators  test-mlogloss-mean  test-mlogloss-std  train-mlogloss-mean  \\\n",
       "995           995            0.598319           0.002226             0.553618   \n",
       "996           996            0.598294           0.002224             0.553553   \n",
       "997           997            0.598272           0.002231             0.553501   \n",
       "998           998            0.598250           0.002230             0.553437   \n",
       "999           999            0.598229           0.002231             0.553381   \n",
       "\n",
       "     train-mlogloss-std  \n",
       "995            0.001038  \n",
       "996            0.001036  \n",
       "997            0.001030  \n",
       "998            0.001030  \n",
       "999            0.001027  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_nestimators = pd.read_csv(\"2_nestimators.csv\")\n",
    "data_nestimators.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_3.get_params()['n_estimators']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在固定最佳的参数max_depth = 5,min_child_weight = 2，默认学习率learning_rate =0.1，重新调节n_estimator，最优的n_estimator = 276，比第一次调优的338小很多，而本轮的最优结果logloss = 0.587131 ，反而也比上一轮的0.583831大了一些，个人猜测：  \n",
    "1、经过以上几轮训练和参数调优，其实参数n_estimator,max_depth和min_child_weight的最优参数都缩小了，而且样本随机采样的比例是0.5，是否选取的样本太少了或这些参数取值太小了造成了当前的模型已经有点欠拟合了  \n",
    "2、默认的学习率learning_rate =0.1，是否学习率太大了，造成训练时在最优解附近震荡，收敛不到最优解\n",
    "\n",
    "尝试将学习率降低为0.01试一下（之前自学过深度学习，其中学习率是一个很重要的参数，一般训练时取值是随迭代次数增加，逐渐减小，这里先不考虑这么多）\n",
    "\n",
    "结果表明，我是作死的[捂脸]，模型训练跑了巨久，但是这里我很不理解为什么步长设置到最小，反而效果更差，logloss = 0.598229，猜测原因是步长设置太小，模型收敛特别慢，而且弱学习器数量这么大，很有可能已经过拟合了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
